NL2024861B1 - Method for recording and correlating light and charged particle microscopy images - Google Patents

Method for recording and correlating light and charged particle microscopy images Download PDF

Info

Publication number
NL2024861B1
NL2024861B1 NL2024861A NL2024861A NL2024861B1 NL 2024861 B1 NL2024861 B1 NL 2024861B1 NL 2024861 A NL2024861 A NL 2024861A NL 2024861 A NL2024861 A NL 2024861A NL 2024861 B1 NL2024861 B1 NL 2024861B1
Authority
NL
Netherlands
Prior art keywords
charged particle
view
image
sample
images
Prior art date
Application number
NL2024861A
Other languages
Dutch (nl)
Inventor
Den Hoedt Sander
Srinivasa Raja Aditi
Irwin Lane Ryan
Pieter Hoogenboom Jacob
Original Assignee
Univ Delft Tech
Delmic Ip B V
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Univ Delft Tech, Delmic Ip B V filed Critical Univ Delft Tech
Priority to NL2024861A priority Critical patent/NL2024861B1/en
Priority to EP21708080.3A priority patent/EP4100985A1/en
Priority to PCT/NL2021/050075 priority patent/WO2021158111A1/en
Application granted granted Critical
Publication of NL2024861B1 publication Critical patent/NL2024861B1/en

Links

Classifications

    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
    • H01J37/00—Discharge tubes with provision for introducing objects or material to be exposed to the discharge, e.g. for the purpose of examination or processing thereof
    • H01J37/02—Details
    • H01J37/22—Optical or photographic arrangements associated with the tube
    • H01J37/226—Optical arrangements for illuminating the object; optical arrangements for collecting light from the object
    • H01J37/228—Optical arrangements for illuminating the object; optical arrangements for collecting light from the object whereby illumination and light collection take place in the same area of the discharge
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
    • H01J37/00—Discharge tubes with provision for introducing objects or material to be exposed to the discharge, e.g. for the purpose of examination or processing thereof
    • H01J37/26—Electron or ion microscopes; Electron or ion diffraction tubes
    • H01J37/28—Electron or ion microscopes; Electron or ion diffraction tubes with scanning beams
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
    • H01J2237/00—Discharge tubes exposing object to beam, e.g. for analysis treatment, etching, imaging
    • H01J2237/22—Treatment of data
    • H01J2237/226—Image reconstruction

Abstract

The invention relates to a method for recording and correlating light and charged particle microscopy images of a sample or one of more sections of a sample, wherein the method comprises the steps of: recording a light microscopy image of a first field of view; recording a first charged particle microscopy image of a second field of view at least partially overlapping with the first field of view, wherein the first charged. particle microscopy image has a first magnification; recording' multiple second. charged. particle microscopy images of third fields of view, wherein each of the multiple second charged particle microscopy images has a second magnification which is higher than the first magnification, and wherein the third fields of view are sub— fields of the second field of view; registering the light microscopy image to the first charged particle microscopy image; and registering the multiple second charged particle microscopy images to the first charged particle microscopy image.

Description

P137251NLO00. Method for recording and correlating light and charged particle microscopy images
BACKGROUND The invention relates to a method for recording and correlating light and charged particle microscopy images.
The aim of large-scale or serial section imaging with an electron microscope is in general to obtain a high- resolution, for example in the range of 1-10 nm, map or three- dimensional volume reconstruction of the structure, for example, of a biological specimen, Such a map or three- dimensional volume construction can be compared to an atlas for the respective biological specimen.
A known method for obtaining a high-resolution map or three-dimensional volume reconstruction involves scanning a sample multiple times by means of an electron microscope.
The first scan is made at low magnification, and the first low magnification scan is used to select regions to be scanned at a higher magnification. These steps are repeated up to the point where one defines the regions for scanning at highest magnification, resulting in a resolution in the range of 1- 10 nm. This method has been applied for imaging the brain of a zebrafish embryo.
Another known method involves the use of a light microscope, wherein the sample to be scanned is equipped with one or more fluorescence labels. The fluorescence labels are arranged at on those regions of the sample, of which, for example, a high-resolution map is required. Additionally, by detecting in-situ whether or not fluorescence is present, it can be decided in an automated fashion which regions of the sample not to scan or to scan in low magnification, and which regions to scan in high magnification. Yet another method involves scanning a sample by means of a so-called multi-beam electron microscope. In such a microscope, a sample is scanned by means of multiple electron beams. As this leads to a higher throughput, it means that the first entire scan of the sample can be conducted at a higher magnification than in the first mentioned method where only a single beam is used.
SUMMARY OF THE INVENTION A disadvantage of the known method that involves scanning a sample multiple times by means of an electron microscope is that the total collection time for scanning the sample up to the point where one defines the regions for scanning at highest magnification is a very time-consuming method. In case of the mentioned example of imaging the brain of a zebrafish embryo, the total collection time including the multiple scans consisted of almost a year. Thus, the throughput of this method is too low in order to collect the required data within an experimentally realistic time. A disadvantage of the method involving the use of a light microscope for obtaining a fluorescence image is that fluorescence and high magnification electron microscopy images have a very different magnification and scale. This results in that several high magnification electron microscopy images have to be recorded and stitched together to fill the field of view of one fluorescence image. Determination of the registration of each of the high magnification electron microscopy images to the fluorescence image is very time-consuming, therewith lowering the throughput of the method. Additionally, in order to achieve a seamless large- scale image, the electron microscopy images have to be stitched together at high precision, preferably comparable to the electron microscope resolution.
The registration accuracy to the fluorescence image is much lower and, therefore, the fluorescence image cannot be used as a template for registration of the electron microscopy images to the fluorescence image.
The registration of the electron microscopy images to the fluorescence image, however, must be maintained during stitching together the electron microscopy images.
In respect of the method including scanning a sample by means of a so-called multi-beam electron microscope, it is noted that a first magnification needs to be selected by the user with which the entire sample is scanned by the multi-beam electron microscope.
This magnification and the corresponding resolution may be limited by the allowed pitch between the individual beams in the respective microscope and the number of pixels per image.
When it is desired to scan specific regions at a higher resolution, it is also required to adjust the magnification of the multi-beam microscope locally on the sample, which may require switching to a single-beam operation.
This reduces the throughput of the multi-beam electron microscope adversely.
To maintain the highest possible throughput, it would be beneficial to select the regions for high magnification acquisition automatically during scanning.
This could be done based on for instance light or fluorescence microscopy images, but then it is again essential to maintain the registration.
It is an object of the present invention toat least partially obviate or eliminate at least one of the disadvantages of the methods mentioned above, or to provide an alternative method for recording and correlating light and charged particle microscopy images.
According to a first aspect, the invention provides a method for recording and correlating light and charged particle microscopy images of a sample or one or more sections of a sample, wherein the method comprises the steps of: — recording a light microscopy image of a first field of view; = recording a first charged particle microscopy image of a second field of view at least partially overlapping with the first field of view, wherein the first charged particle microscopy image has a first magnification; — recording multiple second charged particle microscopy images of third fields of view, wherein each of the multiple second charged particle microscopy images has a second magnification which is higher than the first magnification, and wherein the third fields of view are sub- fields of the second field of view; ~ registering the light microscopy image to the first charged particle microscopy image; and — registering the multiple second charged particle microscopy images to the first charged particle microscopy image.
During applying the method according to the invention, the light microscopy image is registered to the first charged particle microscopy image having the first magnification, which is usually a low magnification.
Furthermore, the multiple second charged particle images having the second magnification, which is usually a high magnification, are registered to the first charged particle microscopy image.
As a result, a registration between the light microscopy image and the multiple second charged particle microscopy images 1s accomplished.
The difference between the magnification of the light microscopy image and the first magnification of the first charged particle microscopy image is relatively small, or at least much smaller than the difference between the magnification of the light microscopy image and each of the multiple second charged particle images.
This same applies for the difference between the first magnification of the first charged particle image and the second magnification of each of the multiple second charged particle microscopy images.
As mentioned above,
determination of the registration of multiple high magnification charged particle microscopy images to a light microscopy image is time-consuming and lowers the throughput.
As each of the second charged particle microscopy images 5 contains similar features in similar contrast, albeit at higher magnification, as part of the first charged particle microscopy image, these can be registered using standard image recognition software.
As a result, determination of the registration for each of the second charged particle microscopy images to the first charged particle microscopy image is less time-consuming then determining the registration of each second charged particle microscopy image to the light microscopy image.
As the time-consuming registration between light and charged particle microscopy images now only has to be conducted once for the low magnification image, the method according to the invention results in a higher throughput.
Additionally, the method has the advantage that the first charged particle microscopy image can be used as a template for stitching together the multiple second charged particle microscopy images.
The registration accuracy to the first charged particle images is sufficient to be used as a template for registration of the multiple second charged particle microscopy images.
Advantageously, the registration Of the second charged particle microscopy images to the light microscopy image can be maintained during stitching together the second charged particle microscopy images via the first charged particle microscopy image.
Moreover, registration of the multiple second charged particle microscopy images with the second magnification to the first charged particle microscopy image with the first magnification may allow for a smaller overlap region between the second charged particle microscopy images.
In the context of the present patent application, this overlap region has to be understood as some extra image area around the second charged particle microscopy images that overlaps with the images of the neighboring third fields of view.
This overlap region is used to stitch together all second magnification images and is usually not seen in the final image. A smaller overlap region results in less area to be scanned, therewith increasing the throughput of the method according to the invention.
In an embodiment, the method comprises the steps of dividing a sample to be imaged into multiple areas, and for each of the multiple areas performing the following steps: - recording a light microscopy image of a first field of view; - recording a first charged particle microscopy image of a second field of view at least partially overlapping with the first field of view, wherein the first charged particle microscopy image has a first magnification; - recording multiple second charged particle microscopy images of third fields of view, wherein each of the multiple second charged particle microscopy images has a second magnification which is higher than the first magnification, and wherein the third fields of view are sub- fields of the second field of view; - registering the light microscopy image to the first charged particle microscopy image; and = registering the multiple second charged particle microscopy images to the first charged particle microscopy image. An advantage of this embodiment is that a sample can be imaged within several repetitions of the given steps, such that it is possible to image the sample completely and at the desired/needed magnification(s).
In an embodiment, the light microscopy images of two or more neighboring areas of the sample to be imaged are recorded before recording the first and second charged particle microscopy images of the respective two or more neighboring areas of the sample to be imaged.
In an embodiment, the first field of view covers the second field of view completely. In an embodiment thereof, the second field of view is smaller than the first field of view, so as to leave one or more borders of the first field of view unexposed to charged particles. During imaging a sample by means of at least one charged particle beam, fluorescence present within the sample will be destroyed by means of charged particles of the at least one charged particle beam. This embodiment has the advantage that, if a section or sample is divided into multiple areas to be imaged, the light microscopy images of the multiple areas can be stitched together by means of the one or more borders of the first field of view unexposed to charged particles.
In an embodiment, when light microscopy images of neighboring areas of the sample to be imaged are recorded, the method comprises the further steps of recording a first charged particle microscopy image of a fourth field of view including one of the one or more borders of the first field of view unexposed to charged particles between the neighboring areas; and recording multiple second charged particle microscopy images of fifth fields of view each including a portion of the one border of the first field of view unexposed to charged particles between the neighboring areas. In an embodiment thereof, the method comprises the step of registering the multiple second charged particle microscopy images of the fifth fields of view with the first charged particle microscopy image of the fourth field of view. An advantage of this embodiment is that first and second charged particle microscopy images are obtained for a complete sample or one of more sections of a sample, while maintaining the possibility to register and stitch together multiple light microscopy images of the sample or the one or more sections of the sample.
In an embodiment, when the method comprises the steps of dividing a sample to be imaged into multiple areas, and for each of the multiple areas performing the following steps: recording a light microscopy image of a first field of view; recording a first charged particle microscopy image of a second field of view at least partially overlapping with the first field of view, wherein the first charged particle microscopy image has a first magnification; recording multiple second charged particle microscopy images of third fields of view, wherein each of the multiple second charged particle microscopy images has a second magnification which is higher than the first magnification, and wherein the third fields of view are sub-fields of the second field of view; registering the light microscopy image to the first charged particle microscopy image; and registering the multiple second charged particle microscopy images to the first charged particle microscopy image, the method comprises the step of stitching together the light microscopy images of the imaged multiple areas of the sample. An advantage of this embodiment is that a large-area composite light microscopy image of a sample or of at least one section of a sample can be obtained.
In an embodiment, the method comprises the step of stitching together the multiple second charged particle microscopy images in order to create a large area composite image, preferably before registering the multiple second charged particle microscopy images to the first charged particle microscopy image.
In a further embodiment thereof, adjacent ones of the multiple second charged particle microscopy images have overlapping parts due to partially overlapping third fields of view, and wherein stitching together the multiple second charged particle microscopy images comprises combining the multiple second charged particle microscopy images with the overlapping parts in order to create the large area composite image. In an embodiment thereof, the method comprises the step of, in order to create the large area composite image, using data of the multiple second charged particle microscopy images, in particular of the overlapping parts thereof, which has been acquired first in time. In the context of the present patent application it is noted that contrast in charged particle, such as electron, images may depend on the total applied charged particle, such as electron, dose. Therefore, each of the overlapping parts of adjacent second charged particle microscopy images used to stitch together the adjacent images has a different contrast. The different contrasts may deteriorate the appearance of the final composite image. By using data of the multiple second charged particle microscopy images, in particular of the overlapping parts thereof, which has been acquired first in time, the contrast differences between different parts of the composite image can be decreased, therewith improving the quality of the composite image and the registration accuracy to the first charged particle microscopy image. Thus, the large area composite image has substantially the same contrast over its complete surface, In an embodiment, the method comprises the steps of determining a first drift of at least one light microscopy beam to a sample from previous light microscopy images, determining a second drift of at least one charged particle beam to a sample from a sequence of charged particle images, and/or using the determined first drift and/or second drift for correcting a registration matrix for drift. The registration matrix is used for registration, for example, of the light microscopy image to the first charged particle microscopy image. This embodiment has the advantage that by including a correction for the drift of the at least one light microscopy beam to the sample into the registration matrix, registration and correction for drift can be performed in a single event.
In an embodiment, the method comprises, before the step of recording a first charged particle microscopy image, the steps of recording one or more high-resolution or super- resolution, preferably fluorescence, light microscopy images of a sixth field of view equivalent to or smaller than the first field of view, and registering the one or more high- resolution or super-resolution images to the light microscopy image. In an embodiment thereof, the method comprises the steps of detecting a predetermined fluorescence signal on a sample, and the step of recording a high-resolution or super- resolution, preferably fluorescence, light microscopy image comprises recording a high-resolution or super-resolution,
preferably fluorescence, light microscopy image of those areas of the sample where the predetermined fluorescence signal has been detected. An advantage of this embodiment is that at least of areas of the sample where the predetermined fluorescence signal has been detected, light microscopy images can be taken with a higher resolution than the resolution imposed by the diffraction limit. As a result, high-resolution light microscopy images of the sample can be taken and, subsequently, registered indirectly to the second charged particle microscopy images having a high magnification.
In an embodiment, light microscopy image(s), first charged particle microscopy image(s) and second charged particle microscopy images are recorded for multiple sections of a sample and/or the method is repeated for one or more following sections. In an embodiment thereof, these sections have been consecutively cut from a 3D sample and thus data acquired for each section of the one or more sections of the sample is used to create a 3D stack of registered, composite images.
In an embodiment, the steps of recording the light microscopy image(s) and of recording the first charged particle microscopy image(s) are performed on an integrated microscope assembly, and the step of recording the multiple second charged particle microscopy images is performed on a multi-beam charged particle system. A multi-beam charged particle microscopy system has the possibility to scan large areas at once, because of the multiple beams. The time required for recording the multiple second charged particle microscopy images can be shortened in comparison with a single beam charged particle microscopy. Thus, this embodiment reduces the time needed for recording and correlating light and charged particles images of a sample or one or more sections of a sample.
In another embodiment, the step of recording the first charged particle microscopy image(s) is performed on a multi-beam scanning electron microscope (SEM) and the step of recording the multiple second charged particle microscopy images is performed on a single-beam SEM, or the multi-beam SEM operating in single-beam mode, or the multi-beam SEM operating in a higher resolution or smaller pitch mode.
In an embodiment, the method comprises the step of determining registration of the light microscopy image to the first charged particle microscopy image by means of a previously determined registration procedure. This allows a further increase in throughput as the time-consuming light to charged particle microscopy registration does not need to be conducted for each pair of first and second or fourth fields of view, but can for instance be calculated from the previously determined registration procedure and the known sample stage or charged particle beam translation coordinates and the determination of the drift. Using a previously determined registration procedure may also allow a light microscope, to be separate from a charged particle microscope. This allows the method according to this embodiment to be performed on conventional light and charged particle microscopes.
According to a second aspect, the invention provides an integrated microscopy assembly for imaging a sample, comprising: - at least one optical microscope configured to observe one or more regions of interest on the sample; - at least one charged particle microscope, such as an ion or electron microscope, to emit and to focus at least one charged particle beam on one or more positions on the sample; and = an image processing device, wherein the image processing device is configured for performing the steps of the method according to any one of the preceding claims.
The integrated microscopy assembly according to the invention has at least the same advantages as described in relation to the method according to the first aspect of the invention. The various aspects and features described and shown in the specification can be applied, individually, wherever possible. These individual aspects, in particular the aspects and features described in the attached dependent claims, can be made subject of divisional patent applications.
BRIEF DESCRIPTION OF THE DRAWINGS The invention will be elucidated on the basis of an exemplary embodiment shown in the attached drawings, in which: Figure 1 shows a schematic overview of an integrated microscopy system according to an embodiment of the invention; Figure 2 shows a schematic overview of a method for recording and imaging light and charged particle microscopy images according to a first embodiment of the invention; Figures 3A and 3B schematically show a top view of a sample to be imaged with indicated a first and second field of view, and thirds fields of view; and a top view of two adjacent recorded charged particle images, respectively; and Figure 4 shows a schematic overview of a method for recording and imaging light and charged particle microscopy images according to a second embodiment of the invention.
DETAILED DESCRIPTION OF THE INVENTION In order to elucidate the context of the present invention, a principle set-up of a so-called optical SEM combination is provided by way of example. With reference to figure 1, the basic design of an integrated microscopy system 1 is explained. It is noted that the shown example of the integrated microscopy system 1 is only included for the purpose of exemplifying the principle of the integrated microscopy system 1 and does not exclude any alternative or unknown variations of such an integrated microscopy system
1. Specifically, the system may be of a non-confocal type, for instance wide-field type, it may include a light beam to excite luminescence in the sample, or it may have multiple individual beams.
Figure 1 shows an integrated microscopy system 1 comprising at least an optical microscope 2, 3, 4 to observe one or more regions of interest on a sample 10 on a non-shown sample holder, and a charged particle microscope, such as an ion or electron microscope 7, 8 to emit and to focus at least one charged particle beam 9 on one or more positions on the sample 10 within the one or more regions of interest on the sample 10.
The charged particle microscope is provided with a charged particle source 7 for emitting a primary charged particle beam 9 and to focus said charged particle beam 9 on the sample 10 within the one or more regions of interest. Further, the charged particle microscope comprises a detector 8 for detection of secondary radiation or secondary charged particles 11 backscattered from the sample 10, or emitted, transmitted, or scattered from the sample 10 and possibly induced by the primary beam 9. The charged particle microscope 7, 8 is arranged within a vacuum chamber 13.
The optical microscope 2, 3, 4 is equipped with a light collecting device 2 to receive in use luminescence light 12 emitted by the sample and/or substrate 10 and induced by the primary beam 9 and to focus the luminescence light on a photon-detector 4. The light collecting device 2 may be an objective, a mirror or a glass fiber. It may also consist of a plurality of devices to arrange for collecting and focusing of the concerning luminescence light 12 that is emitted by the sample 10, e.g. using a known per se CCD camera.
In the present example the optical microscope 2, 3, 4 is of the confocal type having a pinhole 3 between the light collecting device 2 and the photon-detector 4. In this embodiment the optical microscope 2, 3, 4 is placed entirely inside the vacuum chamber 13 of the charged particle microscope 7, 8. The closed dashed line 14 encircles those parts of the integrated microscopy assembly 1 according to the invention that may be mounted on the (replaceable) door of the vacuum chamber 13, notably the sample holder for the sample 10, the light collecting device 2, the pinhole 3, and the photon-detector 4.
As shown in figure 1, the integrated microscopy system 1 is provided with an image processing device 30, alternatively denoted as controller or processor. The image processing device 30 is useable as an automation unit and may be in the form of a computer, including a personal computer provided with dedicated software. The control unit 30 may have with one or more screens, e.g. one screen or screen part for depicting the recorded optical image, and another screen or another part of the same screen depicting an image, in particular of the same sample or substrate, recorded via the charged particle part of the integrated microscopy assembly
1. In general, the image processing device is among others configured for implementing one or more methods of use and controlling the integrated microscopy assembly 1. In particular, the image processing device 30 is configured for performing a method for recording and correlating light and charged particle microscopy images according to an embodiment Of the invention.
Figure 2 shows a schematic overview of the steps of the method for recording and correlating light and charged particle microscopy images according to an embodiment of the invention. As a first step, the method comprises the step of arranging S1 a sample holder with the sample 10 placed thereon within the integrated microscopy assembly 1, for example such as shown in figure 1. Subseguently, the method comprises the step of recording S2, by means of the optical microscope 2, 3, 4, a light microscopy image of a first field of view 35 on the sample 10, as schematically indicated in figure 3A and of which the right border is indicated by means of the dashed line. The recorded light microscopy image is transmitted to the image processing device 30 and, optionally, shown on at least one of the one or more screens thereof. After the light microscopy image has been recorded, the step of recording S3 a first charged particle image of a second field of view 36 on the sample 10 is performed. The first charged particle image is recorded at a first magnification, in particular a low magnification. As schematically indicated in figure 3A, the second field of view 36 overlaps with the first field of view 35 and is slightly smaller than the first field of view 35, such that an unexposed border of fluorescence 37 remains. The unexposed border of fluorescence 37, for example, can be used for stitching adjacent light microscopy images of a set of light microscopy images.
The step of recording S3 a first charged particle image is followed by a step of recording S4 multiple second charged particle microscopy images of third fields of view
38. Each of the multiple second charged particle microscopy images is recorded at a second magnification, in particular a high magnification. In the context of the present patent application, it implies that the higher the magnification, the higher the number of details within the recorded images. The third fields of view 38 are located within the second field of view 36 and are sub-fields of the second field of view 36. The third fields of view 38 may fill the second field of view 36 completely or partially, depending on a region of interest of the sample 10.
After acquiring the images described above, the method comprises the step of registering S5 the light microscope image to the first charged particle image. The difference in magnification of the light microscope image and the magnification of the first charged particle image is relatively small.
Additionally, the method also comprises the step of stitching together S6 the multiple second charged particle images to generate a single high-magnification image, followed by registering S7 the single high-magnification image to the first charged particle image. During stitching together the multiple second charged particle images, adjacent images with overlapping parts 39 due to partially overlapping third fields of view 38, as is schematically indicated in figure 3B, are combined in order to generate the single high-magnification image of the multiple second charged particle images. In order to create the multiple second charged particle images which can be stitched together, the overlapping parts 39 of the third fields of view 38, as indicated in figure 3B, are exposed to the at least one charged particle beam 9 multiple times, usually two times. Preferably the overlapping part 39 of each of the second charged particle images 39 which is exposed to the at least one charged particle beam 9 only once is used for generating the single high-resolution image.
Furthermore, the difference between the magnification of the first charged particle image and the magnification of the single high-magnification image formed from the multiple second charged particle images is relatively small. This contributes advantageously to the registration result of the single high-magnification image, i.e. the stitched together multiple second charged particle images, to the first charged particle image.
As a result of the method according to this embodiment of the invention the stitched together multiple charged particle images are registered to the light microscopy image by means of the first charged particle image with a sufficient registration accuracy. Additionally, since registration of the multiple second charged particle images to the first charged particle image is less time-consuming than registration of the multiple second charged particle images to the light microscopy image, the method according to this embodiment of the invention requires less time for acquiring the registration result.
Optionally, the sample 10 to be imaged has to be imaged in multiple steps in order to cover the complete surface of the sample 10 or to acquire high-resolution images.
In that case, steps S2 —- S57 are repeated for two or more areas, as schematically indicated by the first fields of view 35 in figure 3A, wherein the facing unexposed borders of fluorescence 37 of adjacent first fields of view 35 form a fluorescence overlapping part 40. The fluorescence overlapping part(s) 40 can be used for stitching together all light microscopy images which are made of the sample 10, such that a large-area light microscopy image of the sample 10 can be obtained.
When steps S2 — 87 are completed for adjacent first fields of view 35, the fluorescence overlapping part 40 is imaged by means of the charged particle microscope 7,9. In particular, firstly a first charged particle image of the respective fluorescence overlapping part 40 is recorded, as in step S3, followed by recording, as in step $54, multiple second charged particle images of the respective fluorescence overlapping part 40. The first charged particle image of the respective fluorescence overlapping part 40 is stitched together to the first charged particle images of the adjacent areas in the same manner as described in relation to step 56, in order to realize a single low-magnification charged particle image of the two or more areas. The multiple second charged particle images of the respective fluorescence overlapping part 40 are stitched together and to the multiple second particle images of the one or more adjacent areas, as in step S86, in order to create a single high-magnification charged particle image. Following thereon, the single high- magnification charged particle image is registered to the low-magnification charged particle image and to the large- area light microscopy image. As a result, the single high- magnification charged particle image is registered to the large-area light microscopy image.
Figure 4 shows a schematic overview of the steps of the method for recording and correlating light and charged particle microscopy images according to another embodiment of the invention. The method comprises substantially the same steps as the method schematically indicated in figure 2. For the sake of brevity, refrained is from reintroducing similar steps and, therefore, similar steps are indicated by the same reference number increased with 100.
The method schematically indicated in figure 4 differs from the previous described method in that this method comprises the step of recording S108 one or more high- or super-resolution light microscopy images of at least a part of the sample 10, in particular of those parts of the sample where a predetermined fluorescence signal has been 10 detected in the light microscopy image. It is noted that this step S108 is performed after step $5102 and before step S103. Furthermore, the recorded one or more high- or super- resolution light microscopy images are registered S109 to the light microscopy image(s) and, therewith, to the multiple second charged particle images.
It is noted that the steps of the embodiments of the method according to the invention can be performed on various sections, for example, of a biological specimen to be imaged, wherein each section corresponds to the sample 10 as described above. When each of the various sections is imaged, the recorded and registered images can be used for creating a 3D stack of registered, composite images.
In the context of the present application, it is noted that alternative to the integrated microscopy system 1 as shown in figure 1, the first charged particle image(s) can be acquired by means of a multi-beam scanning electron microscope (SEM), and the second charged particle images can be acquired by means of a single-beam SEM, the multi-beam SEM operating in a single-beam mode, or the multi-beam SEM operating in a higher resolution or smaller pitch mode in comparison with recording the first charged particle image (s).
It is to be understood that the above description is included to illustrate the operation of the preferred embodiments and is not meant to limit the scope of the invention. From the above discussion, many variations will be apparent to one skilled in the art that would yet be encompassed by the scope of the present invention.

Claims (20)

CONCLUSIESCONCLUSIONS 1. Werkwijze voor het opnemen en correleren van licht- en geladen-deeltjesmicroscopie-afbeeldingen van een monster of één of meer secties van een monster, waarbij de werkwijze de stappen omvat van: — het opnemen van een lichtmicroscopie- afbeelding van een eerste gezichtsveld; - het opnemen van een eerste geladen- deeltjesmicroscopie-afbeelding van een tweede gezichtsveld dat ten minste gedeeltelijk overlapt met het eerste gezichtsveld, waarbij de eerste geladen-deeltjesmicroscopie- afbeelding een eerste vergroting heeft; — het opnemen van meerdere tweede geladen- deeltjesmicroscopie-afbeeldingen van derde gezichtsvelden, waarbij elke van de meerdere tweede geladen- deeltjesmicroscopie-afbeeldingen een tweede vergroting heeft die hoger is dan de eerste vergroting, en waarbij de derde gezichtsvelden sub-velden zijn van het tweede gezichtsveld; = het registreren van de lichtmicroscopie- afbeelding met de eerste geladen-deeltjesmicroscopie- afbeelding; en = het registreren van de meerdere tweede geladen- deeltjesmicroscopie-afbeeldingen met de eerste geladen- deeltjesmicroscopie-afbeelding.A method of recording and correlating light and charged particle microscopy images of a sample or one or more sections of a sample, the method comprising the steps of: - taking a light microscopy image of a first field of view; - recording a first charged particle microscopy image of a second field of view at least partially overlapping with the first field of view, the first charged particle microscopy image having a first magnification; — recording multiple second charged particle microscopy images of third fields of view, each of the plurality of second charged particle microscopy images having a second magnification greater than the first magnification, and wherein the third fields of view are sub-fields of the second field of view; = registering the light microscopy image with the first charged particle microscopy image; and = registering the plurality of second charged particle microscopy images with the first charged particle microscopy image. 2. Werkwijze volgens conclusie 1, omvattend de stappen van het opdelen van een af te beelden monster in meerdere gebieden en voor elk van de meerdere gebieden het uitvoeren van de volgende stappen: - het opnemen van een lichtmicroscopie- afbeelding van een eerste gezichtsveld; — het opnemen van een eerste geladen- deeltjesmicroscopie-afbeelding van een tweede gezichtsveld dat ten minste gedeeltelijk overlapt met het eerste gezichtsveld, waarbij de eerste geladen-deeltjesmicroscopie-A method according to claim 1, comprising the steps of dividing a sample to be imaged into a plurality of regions and for each of the plurality of regions performing the following steps: - taking a light microscopy image of a first field of view; — recording a first charged particle microscopy image of a second field of view that at least partially overlaps with the first field of view, wherein the first charged particle microscopy afbeelding een eerste vergroting heeft; — het opnemen van meerdere tweede geladen- deeltjesmicroscopie-afbeeldingen van derde gezichtsvelden, waarbij elke van de meerdere tweede geladen- deeltjesmicroscopie-afbeeldingen een tweede vergroting heeft die hoger is dan de eerste vergroting, en waarbij de derde gezichtsvelden sub-velden zijn van het tweede gezichtsveld; = het registreren van de lichtmicroscopie- afbeelding met de eerste geladen-deeltjesmicroscopie- afbeelding; en = het registreren van de meerdere tweede geladen- deeltjesmicroscopie-afbeeldingen met de eerste geladen- deeltjesmicroscopie-afbeelding.image has a first magnification; — recording multiple second charged particle microscopy images of third fields of view, each of the plurality of second charged particle microscopy images having a second magnification greater than the first magnification, and wherein the third fields of view are sub-fields of the second field of view; = registering the light microscopy image with the first charged particle microscopy image; and = registering the plurality of second charged particle microscopy images with the first charged particle microscopy image. 3. Werkwijze volgens conclusie 2, waarbij de Llichtmicroscopie-afbeeldingen van twee of meer naburige gebieden van het af te beelden monster worden opgenomen voorafgaand aan het opnemen van de eerste en tweede geladen- deeltjesmicroscopie-afbeeldingen van de respectievelijke twee of meer naburige gebieden van het af te beelden monster.The method of claim 2, wherein the L-light microscopy images of two or more adjacent regions of the sample to be imaged are acquired prior to recording the first and second charged particle microscopy images of the respective two or more adjacent regions of the said sample. sample to be displayed. 4, Werkwijze volgens één der voorgaande conclusies, waarbij het eerste gezichtsveld het tweede gezichtsveld compleet bedekt.A method according to any one of the preceding claims, wherein the first field of view completely covers the second field of view. 5. Werkwijze volgens conclusie 4, waarbij het tweede gezichtsveld kleiner is dan het eerste gezichtsveld, om zodoende één of meer randen van het eerste gezichtsveld niet-blootgesteld aan geladen deeltjes te laten.The method of claim 4, wherein the second field of view is smaller than the first field of view, so as to leave one or more edges of the first field of view unexposed to charged particles. 6. Werkwijze volgens conclusie 5, wanneer lichtmicroscopie-afbeeldingen van naburige gebieden van het af te beelden monster zijn opgenomen, omvattend de verdere stappen van het opnemen van een eerste geladen- deeltjesmicroscopie-afbeelding van een vierde gezichtsveld omvattend één van de ene of meerdere randen van het eerste gezichtsveld die niet zijn blootgesteld aan geladen deeltjes, tussen de naburige gebieden; en het opnemen van meerdere tweede geladen-deeltjesmicroscopie-afbeeldingen van vijfde gezichtsvelden die elk een gedeelte omvatten van de ene rand van het eerste gezichtsveld, die niet is blootgesteld aan geladen deeltjes, tussen de naburige gebieden.The method of claim 5, when light microscopy images of adjacent regions of the sample to be imaged are recorded, comprising the further steps of recording a first charged particle microscopy image of a fourth field of view comprising one of the one or more edges of the first field of view not exposed to charged particles, between the neighboring regions; and incorporating a plurality of second charged particle microscopy images of fifth fields of view each comprising a portion of one edge of the first field of view, which is not exposed to charged particles, between the adjacent regions. 7. Werkwijze volgens conclusie 6, omvattend de stap van het registreren van de meerdere tweede geladen- deeltjesmicroscopie-afbeeldingen van de vijfde gezichtsvelden met de eerste geladen-deeltjesmicroscopie- afbeelding van het vierde gezichtsveld.The method of claim 6, comprising the step of registering the plurality of second charged particle microscopy images of the fifth fields of view with the first charged particle microscopy image of the fourth field of view. 3. Werkwijze volgens één der conclusies 2-7, wanneer afhankelijk van conclusie 2, omvattend de stap van het aan elkaar stikken van de lichtmicroscopie-afbeeldingen van de afgebeelde meerdere gebieden van het monster.The method of any one of claims 2 to 7 when dependent on claim 2, comprising the step of stitching together the light microscopy images of the imaged multiple regions of the sample. 9. Werkwijze volgens een der voorgaande conclusies, omvattend de stap van het aan elkaar stikken van de meerdere tweede geladen-deeltjesmicroscopie-afbeeldingen om een samengestelde afbeelding van een groot gebied te creëren, bij voorkeur voorafgaand aan het registreren van de meerdere tweede geladen-deeltjesmicroscopie-afbeeldingen met de eerste geladen-deeltjesmicroscopie-afbeelding.A method according to any preceding claim comprising the step of stitching the plurality of second charged particle microscopy images together to create a composite image of a large area, preferably prior to recording the plurality of second charged particle microscopy images with the first charged particle microscopy image. 10. Werkwijze volgens conclusie 9, waarbij aangrenzende van de meerdere tweede geladen- deeltjesmicroscopie-afbeeldingen overlappende delen hebben als gevolg van gedeeltelijk overlappende derde gezichtsvelden, en waarbij het aan elkaar stikken van de meerdere tweede geladen-deeltjesmicroscopie-afbeeldingen het combineren omvat van de meerdere tweede geladen- deeltjesmicroscopie-afbeeldingen met de overlappende delen voor het creëren van de samengestelde afbeelding van een groot gebied.The method of claim 9, wherein adjacent of the plurality of second charged particle microscopy images have overlapping portions due to partially overlapping third fields of view, and wherein stitching the plurality of second charged particle microscopy images together comprises combining the plurality second charged particle microscopy images with the overlapping portions to create the composite image of a large area. 11. Werkwijzer volgens conclusie 10, omvattend de stap van, teneinde de samengestelde afbeelding van een groot gebied te verkrijgen, het gebruiken van data van de meerdere tweede geladen-deeltjesmicroscopie-afbeeldingen, in het bijzonder van de overlappende delen daarvan, die het eerste in tijd is verkregen.A method according to claim 10, comprising the step of, in order to obtain the composite image of a large area, using data from the plurality of second charged particle microscopy images, in particular from the overlapping portions thereof, which form the first in time is obtained. 12. Werkwijze volgens één der voorgaande conclusies, omvattend de stappen van het bepalen van een eerste drift van ten minste éen lichtmicroscopiebundel ten opzichte van een monster uit eerdere eerste lichtmicroscopie-A method according to any preceding claim comprising the steps of determining a first drift of at least one light microscopy beam relative to a sample from previous first light microscopy samples. afbeeldingen, het bepalen van een tweede drift van ten minste één geladen-deeltjesbundel ten opzichte van een monster uit een sequentie van geladen deeltjesafbeeldingen, en/of het gebruiken van de bepaalde eerste drift en/of tweede drift voor het corrigeren van een registratiematrix voor drift.images, determining a second drift of at least one charged particle beam relative to a sample from a sequence of charged particle images, and/or using the determined first drift and/or second drift to correct a drift registration matrix . 13. Werkwijze volgens één der voorgaande conclusies, omvattend, voorafgaand aan de stap van het opnemen van een eerste geladen-deeltjesmicroscopie- afbeelding, de stappen van het opnemen van één of meer hoge- resolutie of super-resolutie, bij voorkeur fluorescentie, lichtmicroscopie-afbeeldingen van een zesde gezichtsveld dat equivalent is aan of kleiner is dan het eerste gezichtsveld, en het registreren van de ene or meerdere hoge-resolutie of super-resolutie afbeeldingen met de lichtmicroscopie- afbeelding.A method according to any preceding claim comprising, prior to the step of recording a first charged particle microscopy image, the steps of recording one or more high-resolution or super-resolution, preferably fluorescence, light microscopy images of a sixth field of view equivalent to or smaller than the first field of view, and registering the one or more high-resolution or super-resolution images with the light microscopy image. 14. Werkwijze volgens conclusie 13, omvattend de steppen van het detecteren van een vooraf bepaalde fluorescentiesignaal aan een monster, en waarbij de stap van het opnemen van een hoge-resolutie of super-resolutie, bij voorkeur fluorescentie, lichtmicroscopie-afbeeldingen het opnemen omvat van een hoge-resolutie of super-resolutie, bij voorkeur fluorescentie, lichtmicroscopie-afbeelding van die gebieden van het monster waar het vooraf bepaalde fluorescentiesignaal is gedetecteerd.The method of claim 13, comprising the steps of detecting a predetermined fluorescence signal on a sample, and wherein the step of recording high-resolution or super-resolution, preferably fluorescence, light microscopy images comprises recording a high-resolution or super-resolution, preferably fluorescence, light microscopy image of those regions of the sample where the predetermined fluorescence signal is detected. 15. Werkwijze volgens één der voorgaande conclusies, waarbij lichtmicroscopie-afbeelding{en), eerste geladen-deeltjesmicroscopie-afbeelding (en) en tweede geladen-deeltjesmicroscopie-afbeelding(en) worden opgenomen voor meerdere secties van een monster en/of waarbij de werkwijze wordt herhaald voor één of meer volgende secties.The method of any preceding claim, wherein light microscopy image(s), first charged particle microscopy image(s) and second charged particle microscopy image(s) are recorded for multiple sections of a sample and/or wherein the method is repeated for one or more subsequent sections. 16. Werkwijze volgens conclusie 15, waarbij deze secties opeenvolgend zijn gesneden van een 3D-monster en waarbij dus data die is verkregen voor elke sectie van de ene of meerdere secties van het monster wordt gebruikt voor het creëren van een 3D-stapel van geregistreerde, samengestelde afbeeldingen.The method of claim 15, wherein said sections are sequentially cut from a 3D sample and thus data obtained for each section from the one or more sections of the sample is used to create a 3D stack of recorded, composite images. 17. Werkwijze volgens één der voorgaande conclusies, waarbij de stappen van het opnemen van de lichtmicroscopie-afbeelding{en} en van het opnemen van de eerste geladen-deeltjesmicroscopie-afbeelding{en) worden uitgevoerd op een geïntegreerd microscoopsamenstel, en waarbij de stap van het opnemen van de meerdere tweede geladen -deeltjesmicroscopie-afbeeldingen wordt uitgevoerd op een geladen-deeltjessysteem met meerdere bundels.The method of any preceding claim, wherein the steps of recording the light microscopy image(s) and of recording the first charged particle microscopy image(s) are performed on an integrated microscope assembly, and wherein the step of the recording of the multiple second charged particle microscopy images is performed on a multi-beam charged particle system. 18. Werkwijze volgens éen der voorgaande conclusies, waarbij de stap van het opnemen van de eerste geladen-deeltjesmicroscopie-afbeelding{en) wordt uitgevoerd op een scannende elektronenmicroscoop met meerdere bundels (SEM) en de stap van het opnemen van de meerdere tweede geladen-deeltjesmicroscopie-afbeeldingen wordt uitgevoerd op een SEM met een enkele bundel, of de SEM met meerdere bundels die werkzaam is in een enkele-bundelmodus, of de SEM met meerdere bundels die werkzaam is in een modus met hogere resolutie of kleinere afstand.The method of any preceding claim, wherein the step of recording the first charged particle microscopy image(s) is performed on a multi-beam scanning electron microscope (SEM) and the step of recording the plurality of second charged image(s). particle microscopy imaging is performed on a single-beam SEM, or the multi-beam SEM operating in a single-beam mode, or the multi-beam SEM operating in a higher resolution or closer range mode. 19. Werkwijze volgens één der voorgaande conclusies, omvattend de stap van het bepalen van registratie van de lichtmicroscopie-afbeelding met de eerste geladen- deeltjesmicroscopie-afbeelding door middel van een eerder bepaalde registratieprocedure.The method of any preceding claim, comprising the step of determining registration of the light microscopy image with the first charged particle microscopy image by a previously determined registration procedure. 20. Geïntegreerd microscoopsamenstel voor het afbeelden van een monster, omvattend: — ten minste een optische microscoop die is geconfigureerd voor het observeren van één of meer gebieden van belang van het monster; — ten minste één geladen-deeltjesmicroscoop, zoals een ionen- of elektronenmicroscoop, voor het emitteren en voor het focusseren van Len minste één geladen- deeltjesbundel op één of meer posities op het monster; en - een afbeeldingsverwerkingsinrichting, waarbij de afbeeldingsverwerkingsinrichting is geconfigureerd voor het uitvoeren van de stappen van de werkwijze volgens één der voorgaande conclusies.An integrated microscope assembly for imaging a sample, comprising: - at least one optical microscope configured to observe one or more regions of interest of the sample; — at least one charged particle microscope, such as an ion or electron microscope, for emitting and for focusing Len at least one charged particle beam at one or more positions on the sample; and - an image processing device, wherein the image processing device is configured to perform the steps of the method according to any one of the preceding claims. -0-0-0--0-0-0-0-
NL2024861A 2020-02-07 2020-02-07 Method for recording and correlating light and charged particle microscopy images NL2024861B1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
NL2024861A NL2024861B1 (en) 2020-02-07 2020-02-07 Method for recording and correlating light and charged particle microscopy images
EP21708080.3A EP4100985A1 (en) 2020-02-07 2021-02-05 Method for recording and correlating light and charged particle microscopy images
PCT/NL2021/050075 WO2021158111A1 (en) 2020-02-07 2021-02-05 Method for recording and correlating light and charged particle microscopy images

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
NL2024861A NL2024861B1 (en) 2020-02-07 2020-02-07 Method for recording and correlating light and charged particle microscopy images

Publications (1)

Publication Number Publication Date
NL2024861B1 true NL2024861B1 (en) 2021-09-13

Family

ID=70155270

Family Applications (1)

Application Number Title Priority Date Filing Date
NL2024861A NL2024861B1 (en) 2020-02-07 2020-02-07 Method for recording and correlating light and charged particle microscopy images

Country Status (3)

Country Link
EP (1) EP4100985A1 (en)
NL (1) NL2024861B1 (en)
WO (1) WO2021158111A1 (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040217297A1 (en) * 2000-12-01 2004-11-04 Yeda Research And Development Co. Ltd. Device and method for the examination of samples in a non vacuum environment using a scanning electron microscope
US20100091362A1 (en) * 2008-10-15 2010-04-15 Kabushiki Kaisha Topcon Complex type microscopic device
US20180053627A1 (en) * 2011-05-13 2018-02-22 Fibics Incorporated Microscopy imaging method and system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040217297A1 (en) * 2000-12-01 2004-11-04 Yeda Research And Development Co. Ltd. Device and method for the examination of samples in a non vacuum environment using a scanning electron microscope
US20100091362A1 (en) * 2008-10-15 2010-04-15 Kabushiki Kaisha Topcon Complex type microscopic device
US20180053627A1 (en) * 2011-05-13 2018-02-22 Fibics Incorporated Microscopy imaging method and system

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
NALAN LIV ET AL: "Simultaneous Correlative Scanning Electron and High-NA Fluorescence Microscopy", PLOS ONE, vol. 8, no. 2, 8 February 2013 (2013-02-08), pages e55707, XP055666091, DOI: 10.1371/journal.pone.0055707 *
SCHORB MARTIN ET AL: "New hardware and workflows for semi-automated correlative cryo-fluorescence and cryo-electron microscopy/tomography", JOURNAL OF STRUCTURAL BIOLOGY, ACADEMIC PRESS, UNITED STATES, vol. 197, no. 2, 28 June 2016 (2016-06-28), pages 83 - 93, XP029894438, ISSN: 1047-8477, DOI: 10.1016/J.JSB.2016.06.020 *
VAN DRIEL L F ET AL: "Tools for correlative cryo-fluorescence microscopy and cryo-electron tomography applied to whole mitochondria in human endothelial cells", EUROPEAN JOURNAL OF CELL BIOLOGY, WISSENSCHAFLICHE VERLAGSGESELLSCHAFT, STUTTGART, DE, vol. 88, no. 11, 1 November 2009 (2009-11-01), pages 669 - 684, XP026585829, ISSN: 0171-9335, [retrieved on 20090902], DOI: 10.1016/J.EJCB.2009.07.002 *

Also Published As

Publication number Publication date
EP4100985A1 (en) 2022-12-14
WO2021158111A1 (en) 2021-08-12

Similar Documents

Publication Publication Date Title
US8314837B2 (en) System and method for imaging with enhanced depth of field
US20110090327A1 (en) System and method for imaging with enhanced depth of field
EP1762878B1 (en) Observation apparatus and observation method
US7885447B2 (en) Image acquiring apparatus including macro image acquiring and processing portions, image acquiring method, and image acquiring program
US8440969B2 (en) Method and apparatus for acquiring simultaneous and overlapping optical and charged particle beam images
JP7252190B2 (en) System for generating enhanced depth-of-field synthetic 2D images of biological specimens
US20110091125A1 (en) System and method for imaging with enhanced depth of field
EP2834832B1 (en) Integrated optical and charged particle inspection apparatus
EP2976747B1 (en) Image quality assessment of microscopy images
US9378921B2 (en) Integrated optical and charged particle inspection apparatus
JP3813798B2 (en) electronic microscope
NL2013262B1 (en) Method for inspecting a sample using an assembly comprising a scanning electron microscope and a light microscope.
AU2014236055A1 (en) Referencing in multi-acquisition slide imaging
US7709773B2 (en) Scanning optical device which sets hardware in association with a depth of a focus position based on hardware set values stored in association with depths of focus positions
JP6474409B2 (en) Method for calibrating a laser deflection apparatus of a laser microdissection system and a laser microdissection system
US11243389B2 (en) Optical scanning arrangement and method
JP6131568B2 (en) Microscope device and image forming method
JP5963487B2 (en) Laser microscope and observation method
NL2024861B1 (en) Method for recording and correlating light and charged particle microscopy images
EP2263070B1 (en) Synchronized imaging by way of an optical method and atomic force microscopy
JP4083193B2 (en) electronic microscope
US20220335580A1 (en) Method for producing a corrected image and apparatus
JP4415839B2 (en) X-ray image synthesis method and X-ray analyzer
WO2018131173A1 (en) Image processing device, microscope system, image processing method, and program
JP2010113972A (en) Scanning transmission electron microscope

Legal Events

Date Code Title Description
PD Change of ownership

Owner name: DELMIC IP B.V.; NL

Free format text: DETAILS ASSIGNMENT: CHANGE OF OWNER(S), ASSIGNMENT; FORMER OWNER NAME: TECHNISCHE UNIVERSITEIT DELFT

Effective date: 20230118