CN108038874A - Towards the real-time registration apparatus of scanning electron microscope image and method of sequence section - Google Patents
Towards the real-time registration apparatus of scanning electron microscope image and method of sequence section Download PDFInfo
- Publication number
- CN108038874A CN108038874A CN201711248908.4A CN201711248908A CN108038874A CN 108038874 A CN108038874 A CN 108038874A CN 201711248908 A CN201711248908 A CN 201711248908A CN 108038874 A CN108038874 A CN 108038874A
- Authority
- CN
- China
- Prior art keywords
- mrow
- image
- corresponding points
- msubsup
- msub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000001000 micrograph Methods 0.000 title claims abstract description 18
- 238000004364 calculation method Methods 0.000 claims abstract description 21
- 238000005457 optimization Methods 0.000 claims abstract description 14
- 239000000284 extract Substances 0.000 claims description 9
- 238000000605 extraction Methods 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 8
- 239000011159 matrix material Substances 0.000 claims description 7
- 238000003384 imaging method Methods 0.000 abstract description 10
- 238000012545 processing Methods 0.000 description 7
- 210000004126 nerve fiber Anatomy 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 230000006855 networking Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 210000004556 brain Anatomy 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000010894 electron beam technology Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 241001282153 Scopelogadus mizolepis Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 210000000653 nervous system Anatomy 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- 230000035790 physiological processes and functions Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000000946 synaptic effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/38—Concurrent instruction execution, e.g. pipeline or look ahead
- G06F9/3818—Decoding for concurrent execution
- G06F9/3822—Parallel decoding, e.g. parallel decode units
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30016—Brain
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The present invention relates to scanning electron microscope image registration field, and in particular to a kind of real-time registration apparatus of scanning electron microscope image towards sequence section and method, in order to improve the real-time of sem image registration.The real-time registration apparatus of the present invention includes FPGA and calculation server;Calculation server includes CPU and GPU.FPGA obtains sequence section view data in real time for being directly connected to Electronic Speculum, and calculates the corresponding points between contiguous slices image, finally sends the corresponding points information between the view data obtained from Electronic Speculum and contiguous slices to calculation server;CPU in calculation server, optimization is adjusted to the corresponding points position matched in sequence section;GPU in calculation server, image deformation is carried out according to the corresponding points position after adjustment.The present invention can form the high accuracy to Electronic Speculum system high throughput view data, the long sequence registration ability of low delay, and meet the imaging of high throughput Electronic Speculum sequence section matches somebody with somebody quasi need in real time.
Description
Technical field
The present invention relates to scanning electron microscope image registration field, and in particular to a kind of scanning electron microscope image towards sequence section
Real-time registration apparatus and method.
Background technology
Brain connection collection of illustrative plates research from macroscopic view, Jie's sight and micro-scale by building nervous system structures, and and physiological function
Uniformity understand the operation principle of brain, wherein micro-scale connection collection of illustrative plates is directed to obtaining neuron and cynapse etc. fine
The connection network of structure.The blur-free imaging of synaptic structure is necessarily dependent upon observation method-electron microscope of nanoscale, so as to
In the nervous process tie line that most faint (20~30 nanometers) are followed the trail of in intensive neuropilem.
The neuromechanism connection network of micro-scale is generally obtained by the scanning electron microscope image three-dimensional reconstruction of sequence section
Its three-dimensional appearance.Can not fast and stable obtain the high-resolution three-dimension sem image storehouse of nerve fiber, it is a wide range of prominent to become foundation
One of limitation bottleneck of horizontal neutral net is touched, but lacks effective total solution both at home and abroad.
At present, existing sequence section sem image registration Algorithm is all processed offline sequence section image, i.e., all
Sequence section image carries out image registration work again after all gathering.It is so not only no to utilize what is consumed during Image Acquisition
The magnanimity time, causes to obtain the increase of nerve fiber three-dimensional data required calculating time, and can not in real time see and adopt
Collection finishes the registration result of data, to instruct follow-up collection process, avoids insignificant sequence section micro-image
Collecting work.Therefore, no matter from functional requirement or calculate in view of the time, for big scale of construction nerve fiber sequence section Electronic Speculum
For image, development high throughput real time sequence section sem image registration technique, there is very important theory significance and practicality
Value.
The content of the invention
In order to solve the above problem of the prior art, the present invention proposes a kind of scanning electron microscope (SEM) photograph towards sequence section
, can be complete in real time while nerve fiber sequence section sem image registration accuracy is ensured as real-time registration apparatus and method
Work into registration.
An aspect of of the present present invention, proposes a kind of real-time registration apparatus of scanning electron microscope image towards sequence section, including:
FPGA (Field-Programmable Gate Array, field programmable gate array) and calculation server;
The FPGA, is configured to:Receive the slice image data from scanning electron microscope, and extract current slice image with it is upper
Corresponding points between one sectioning image, described between the current slice view data and the image and a upper sectioning image
The positional information of corresponding points is sent to the calculation server;
The calculation server, including:CPU (Central Processing Unit, central processing unit) and GPU
(Graphics Processing Unit, image processor);
The CPU, is configured to:Often receive the institute between a width slice image data and the image and a upper sectioning image
Corresponding dot position information is stated, just all positions for having received the corresponding points on image are once adjusted, are optimized
The position of the corresponding points afterwards;
The GPU, is configured to:After CPU often completes the once adjustment to the corresponding points, the GPU is all in accordance with excellent
All images that received are carried out a deformation, so as to complete to have received image to all by the position of the corresponding points after change
It is once registering.
Preferably, " slice image data from scanning electron microscope is received, and extracts current slice image and a upper slice map
Corresponding points as between ", specifically include:
View data is received line by line from the scanning electron microscope and is cached, and SIFT feature is extracted according to the data of caching, until
Receive the complete current slice image and complete SIFT feature extraction;
The SIFT feature of the current slice image is matched with the SIFT feature of a upper sectioning image,
Obtain the corresponding points between the current slice image and a upper sectioning image;
Wherein, the line number of caching image data, determines according to the Size of Neighborhood for calculating SIFT feature.
Preferably, " often receive described corresponding between a width slice image data and the image and a upper sectioning image
Dot position information, just once adjusts all positions for having received the corresponding points on image, the institute after being optimized
State the position of corresponding points ", be specially:
Shown method according to the following formula, calculates corresponding when energy function E (w) is minimum valueValue:
And then by all corresponding points received on imagePosition adjustment for optimization after position
Wherein,
I is the sequence number of sequence section image;L is the sequence number of the current slice image;
c1=N (i-1, i) represents the quantity of the corresponding points between the i-th -1 sectioning image and i-th of sectioning image;It is right
In the 1st section, there is c1=0;
c2=N (i, i+1) represents the quantity of the corresponding points between i-th of sectioning image and i+1 sectioning image;It is right
Cut into slices in current l-th, since the L+1 section does not obtain also, there is c2=0;
K and l is the sequence number of corresponding points described in sequence section image, and k ≠ l;
For the position coordinates of k-th of corresponding points in i-th of sectioning image;
For the position coordinates of l-th of corresponding points in i-th of sectioning image;
Respectively position coordinatesMotion vector;
α and β is constant.
Preferably, rapid solving is carried out to min (E (w)) using multi-grid method.
Preferably,
Deformation is carried out to image using Moving Least Squares method on GPU.
Preferably, " after CPU often completes the once adjustment to the corresponding points, the GPU is all in accordance with the institute after optimization
The position of corresponding points is stated, a deformation is carried out to all images that received, so as to complete to have received once matching somebody with somebody for image to all
It is accurate ", be specially:
Different point v is chosen on the i-th width imagei, and corresponding rigid transformation matrix is calculated as follows
And then calculate the position coordinates after deformationUntil point all on the i-th width sectioning image completes deformation;
I=1,2 ..., L are taken, a deformation is carried out as stated above to all images that received, so as to complete to institute
Have and received the once registering of image;
Wherein,
viFor the position at preceding any point of deformation in i-th of sectioning image;
For viRigid transformation matrix to be calculated on point;For viPosition of the point after deformation;
It is corresponding pointsPosition after adjustment;
γ is constant;
Represent corresponding pointsBy rigid transformationPosition coordinates afterwards.
Preferably, each different point v of parallel computation on the GPUiCorresponding rigid transformation matrixAfter deformation
Position
Preferably, the FPGA is connected with the scanning electron microscope and the calculation server respectively by gigabit networking.
Another aspect of the present invention, proposes a kind of scanning electron microscope image Real-time Registration towards sequence section, is based on
The real-time registration apparatus of scanning electron microscope image recited above towards sequence section, comprises the following steps:
Step S10, receives the slice image data from scanning electron microscope, and extracts current slice image and a upper slice map
Corresponding points as between;
Step S20, according to all images having been received by and the corresponding points of extraction, has received on image to all
The position of the corresponding points is adjusted, the position of the corresponding points after being optimized;
All images that received according to the position of the corresponding points after optimization, are carried out deformation, so that complete by step S30
Paired all registrations for having received image;
Step S40, judges whether to have received last width sectioning image, otherwise, goes to step S10.
Preferably, the corresponding points between current slice image and a upper sectioning image are extracted described in step S10, specifically
For:
Calculate the SIFT feature of the current slice image, and the upper sectioning image with being buffered in FPGA
SIFT feature is matched, and obtains the corresponding points between a upper sectioning image.
Preferably, after step slo, before step S20, further include:
Step S15, removes the SIFT feature for the upper sectioning image being buffered in FPGA, by the current slice
The SIFT feature of image is cached, and is used for next sectioning image Feature Points Matching.
Beneficial effects of the present invention:
The real-time registration apparatus of scanning electron microscope image proposed by the present invention towards sequence section and method, based on FPGA+
The Heterogeneous Computing technology of CPU/GPU, and application sequence sectioning image registration Algorithm.The device and method takes full advantage of FPGA's
The parallel processing capability of flowing structure and GPU, can while nerve fiber sequence section sem image registration accuracy is ensured
Meet the imaging of high throughput Electronic Speculum sequence section match somebody with somebody quasi need in real time, realizes the registering in collection of sectioning image, solve from
Line registration causes the problem that 3-dimensional image seriously lags.
Brief description of the drawings
Fig. 1 is that the embodiment of registration apparatus in real time of the invention forms schematic diagram;
Fig. 2 is the image acquisition mode schematic diagram of scanning electron microscope;
Fig. 3 is the flow diagram of Real-time Registration embodiment of the present invention.
Embodiment
The preferred embodiment of the present invention described with reference to the accompanying drawings.It will be apparent to a skilled person that this
A little embodiments are only used for explaining the technical principle of the present invention, it is not intended that limit the scope of the invention.
The real-time registration apparatus of scanning electron microscope image proposed by the present invention towards sequence section, as shown in Figure 1, key is
Start with from hardware-accelerated link, each link of sequence section registration Algorithm is realized on different hardware platforms, builds and is based on
The real-time registration arrangement of Heterogeneous Computing of FPGA+CPU/GPU, and application sequence sectioning image registration Algorithm, realize that Electronic Speculum side gathers
Side is registering in real time.Scanning electron microscope is first imaged sequence section, and during slice imaging, Electronic Speculum passes through kilomega network
View data is sent to FPGA by network in real time, and is completed on FPGA the corresponding points between the section and a upper sectioning image and carried
Take, corresponding points extraction result and original image are sent to calculation server by gigabit networking again.Calculation server stores institute
There are the corresponding points between imaging slice and its contiguous slices, the corresponding points of current slice are added to all imaging slices
In corresponding points, the optimization of corresponding points position is carried out on the CPU of calculation server.According to corresponding points position optimization as a result,
Deformation is carried out to current all images for having received section on the GPU of calculation server.
A kind of embodiment of the real-time registration apparatus of scanning electron microscope image towards sequence section of the present invention, including one piece
FPGA processing boards and a calculation server.FPGA is connected by gigabit networking with scanning electron microscope, calculation server.FPGA is received
View data from scanning electron microscope, sends calculation server to again after processing.Calculation server includes CPU and GPU etc. and calculates
Unit.
FPGA is configured to:Receive the slice image data from scanning electron microscope, and extract current slice image with it is upper all
Corresponding points between picture, by the corresponding points between current slice view data and the image and a upper sectioning image
Positional information is sent to calculation server;
CPU is configured to:It is described right between a width slice image data and the image and a upper sectioning image often to receive
Dot position information is answered, is just once adjusted all positions for having received the corresponding points on image, the correspondence after being optimized
The position of point;
GPU is configured to:After CPU often completes the once adjustment to the corresponding points, after the GPU is all in accordance with optimization
All images that received are carried out a deformation by the position of corresponding points, so as to complete to have received the once registering of image to all.
In the present embodiment, " receive the slice image data from scanning electron microscope, and extract current slice image with it is upper all
Corresponding points between picture ", specifically include:
View data is received line by line from the scanning electron microscope and is cached, and SIFT feature is extracted according to the data of caching, until
Receive the complete current slice image and complete SIFT feature extraction;
The SIFT feature of current slice image is matched with the SIFT feature of a upper sectioning image, is obtained current
Corresponding points between sectioning image and a upper sectioning image;
Wherein, the line number of caching image data, determines according to the Size of Neighborhood for calculating SIFT feature.
Electronic Speculum often completes piece image the imaging of data line, is just sent to the row data in imaging process
FPGA processing, without being retransmited when entire image imaging finishes.Since the imaging mode of scanning electron microscope is electron beam
Electronic signal is inspired in sample surfaces, and the signal strength in electron beam scanning region is obtained using detector, finally line by line
View data is obtained, as shown in Figure 2.This mode is very suitable for being handled using the flowing structure of FPGA.Using which
Electron microscopic data is handled, without waiting for complete finishing image scanning, when can be substantially improved that registration is full-range in real time
Between utilization rate.
The calculating of SIFT feature needs the neighborhood information of current position, thus only a line view data when, it is impossible to count
SIFT feature is calculated, it is necessary to cache some row data in FPGA, then start to calculate, the line number of caching image data is by calculating SIFT
The Size of Neighborhood of feature determines.
In the present embodiment, " often receive described between a width slice image data and the image and a upper sectioning image
Corresponding dot position information, just once adjusts all positions for having received the corresponding points on image, after obtaining optimization
The corresponding points position ", be specially:
According to the method shown in formula (1), calculate corresponding when energy function E (w) is minimum valueValue:
And then by all corresponding points received on sectioning imagePosition adjustment for optimization after position
Wherein,
I is the sequence number of sequence section image;L is the sequence number of current slice image;
c1=N (i-1, i) represents the quantity of corresponding points between the i-th -1 sectioning image and i-th of sectioning image;For
1 section, there is c1=0;
c2=N (i, i+1) represents the quantity of corresponding points between i-th of sectioning image and i+1 sectioning image;For working as
Preceding l-th section, since the L+1 section does not obtain also, there is c2=0;
K and l is the sequence number of corresponding points in sequence section image, and k ≠ l;
For the position coordinates of k-th of corresponding points in i-th of sectioning image;
For the position coordinates of l-th of corresponding points in i-th of sectioning image;
Respectively position coordinatesMotion vector;
E (w) is the energy function of motion vector w;
α and β is constant.
Rapid solving is carried out to min (E (w)) using multi-grid method.
In the present embodiment, deformation is carried out to image using Moving Least Squares method on GPU.
Specifically, different point v is chosen on the i-th width imagei, and calculate corresponding rigid transformation matrix by formula (2)
And then calculate the position coordinates after deformationUntil point all on the i-th width sectioning image completes deformation;
I=1,2 ..., L are taken, a deformation is carried out as stated above to all images that received, so as to complete to institute
Have and received the once registering of image;
Wherein,
viFor the position at preceding any point of deformation in i-th of sectioning image;For viRigid transformation square to be calculated on point
Battle array;For viPosition of the point after deformation;It is corresponding pointsPosition after adjustment;γ
For constant;Represent corresponding pointsBy rigid transformationPosition coordinates afterwards.
Due to every bit v in imageiRigid transformationCalculating all independently carry out, therefore can utilize GPU it is parallel
Calculate, reduce total calculating time.
According to described above, cannot be corresponded to when the registration apparatus rigid connection of the present embodiment receives the 1st width image
Point extraction, corresponding points adjustment, and image deformation;But since the 2nd width image, the new sectioning image of a width is often received,
Extraction with regard to carrying out corresponding points for the width new images, and all images having been received by are subjected to a corresponding points adjustment respectively
With an image deformation, this way, ensure that the image received from Electronic Speculum can carry out registration in real time, be shown as three-dimensional
Image.
The embodiment of a kind of scanning electron microscope image Real-time Registration towards sequence section of the present invention, based on institute above
The real-time registration apparatus of the scanning electron microscope image towards sequence section stated, as shown in figure 3, comprising the following steps:
Step S10, receives the slice image data from scanning electron microscope, and extracts current slice image and a upper slice map
Corresponding points as between;
Step S20, according to all images having been received by and the corresponding points of extraction, has received on image to all
The position of the corresponding points is adjusted, the position of the corresponding points after being optimized;
All images that received according to the position of the corresponding points after optimization, are carried out deformation, so that complete by step S30
Paired all registrations for having received image;
Step S40, judges whether to have received last width sectioning image, if so, then terminating registration task;Otherwise, turn
To step S10.
The corresponding points between current slice image and a upper sectioning image, tool are extracted in the present embodiment, described in step S10
Body is:
Calculate the SIFT feature of the current slice image, and the upper sectioning image with being buffered in FPGA
SIFT feature is matched, and obtains the corresponding points between a upper sectioning image.
In the present embodiment, after step slo, before step S20, further include:
Step S15, removes the SIFT feature for the upper sectioning image being buffered in FPGA, by the current slice
The SIFT feature of image is cached, and is used for next sectioning image Feature Points Matching.
Those skilled in the art should be able to recognize that, each exemplary dress described with reference to the embodiments described herein
Put and method and step, can be realized with electronic hardware, computer software or the combination of the two, in order to clearly demonstrate electronics
The interchangeability of hardware and software, generally describes each exemplary composition and step according to function in the above description
Suddenly.These functions are performed with electronic hardware or software mode actually, and the application-specific and design depending on technical solution are about
Beam condition.Those skilled in the art can realize described function to each specific application using distinct methods, but
It is this realization it is not considered that beyond the scope of this invention.
So far, the preferred embodiment shown in the drawings technical solution that the invention has been described, still, this area are had been combined
Technical staff is it is easily understood that protection scope of the present invention is expressly not limited to these embodiments.Without departing from this
On the premise of the principle of invention, those skilled in the art can make correlation technique feature equivalent change or replacement, these
Technical solution after changing or replacing it is fallen within protection scope of the present invention.
Claims (11)
- A kind of 1. real-time registration apparatus of scanning electron microscope image towards sequence section, it is characterised in that including:FPGA and calculating take Business device;The FPGA, is configured to:Receive the slice image data from scanning electron microscope, and extract current slice image with it is upper all Corresponding points between picture, will be described corresponding between the current slice view data and the image and a upper sectioning image The positional information of point is sent to the calculation server;The calculation server, including:CPU and GPU;The CPU, is configured to:It is described right between a width slice image data and the image and a upper sectioning image often to receive Dot position information is answered, just all positions for having received the corresponding points on image are once adjusted, after being optimized The position of the corresponding points;The GPU, is configured to:After CPU often completes the once adjustment to the corresponding points, after the GPU is all in accordance with optimization The corresponding points position, to it is all received images carry out a deformation, so as to complete to have received the one of image to all Secondary registration.
- 2. real-time registration apparatus according to claim 1, it is characterised in that " receive the sectioning image from scanning electron microscope Data, and extract the corresponding points between current slice image and a upper sectioning image ", specifically include:View data is received line by line from the scanning electron microscope and is cached, SIFT feature is extracted according to the data of caching, until receiving To the complete current slice image and complete SIFT feature extraction;The SIFT feature of the current slice image is matched with the SIFT feature of a upper sectioning image, is obtained Corresponding points between the current slice image and a upper sectioning image;Wherein, the line number of caching image data, determines according to the Size of Neighborhood for calculating SIFT feature.
- 3. real-time registration apparatus according to claim 1, it is characterised in that " often receive a width slice image data with As soon as and the corresponding dot position information between the image and upper sectioning image, to all corresponding points received on image Position once adjusted, the position of the corresponding points after being optimized ", be specially:Shown method according to the following formula, calculates corresponding when energy function E (w) is minimum valueValue:<mfenced open = "" close = ""> <mtable> <mtr> <mtd> <mrow> <mi>min</mi> <mrow> <mo>(</mo> <mrow> <mi>E</mi> <mrow> <mo>(</mo> <mi>w</mi> <mo>)</mo> </mrow> </mrow> <mo>)</mo> </mrow> <mo>=</mo> <mi>min</mi> <mo>(</mo> <munderover> <mi>&Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </munderover> <munderover> <mi>&Sigma;</mi> <mrow> <mi>k</mi> <mo>=</mo> <mn>1</mn> </mrow> <msub> <mi>c</mi> <mn>1</mn> </msub> </munderover> <mo>|</mo> <mo>|</mo> <msubsup> <mi>p</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>+</mo> <msubsup> <mi>w</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>-</mo> <msubsup> <mi>p</mi> <mi>k</mi> <mrow> <mi>i</mi> <mo>+</mo> <mn>1</mn> </mrow> </msubsup> <mo>-</mo> <msubsup> <mi>w</mi> <mi>k</mi> <mrow> <mi>i</mi> <mo>+</mo> <mn>1</mn> </mrow> </msubsup> <mo>|</mo> <msubsup> <mo>|</mo> <mn>2</mn> <mn>2</mn> </msubsup> </mrow> </mtd> </mtr> <mtr> <mtd> <mrow> <mo>+</mo> <mi>&alpha;</mi> <munderover> <mi>&Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>L</mi> </munderover> <munderover> <mi>&Sigma;</mi> <mrow> <mi>k</mi> <mo>=</mo> <mn>1</mn> </mrow> <mrow> <msub> <mi>c</mi> <mn>1</mn> </msub> <mo>+</mo> <msub> <mi>c</mi> <mn>2</mn> </msub> </mrow> </munderover> <munderover> <munder> <mi>&Sigma;</mi> <mrow> <mi>l</mi> <mo>=</mo> <mn>1</mn> </mrow> </munder> <mrow> <mi>l</mi> <mo>&NotEqual;</mo> <mi>k</mi> </mrow> <mrow> <msub> <mi>c</mi> <mn>1</mn> </msub> <mo>+</mo> <msub> <mi>c</mi> <mn>2</mn> </msub> </mrow> </munderover> <mfrac> <mrow> <mo>|</mo> <mo>|</mo> <msubsup> <mi>w</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>-</mo> <msubsup> <mi>w</mi> <mi>l</mi> <mi>i</mi> </msubsup> <mo>|</mo> <msubsup> <mo>|</mo> <mn>2</mn> <mn>2</mn> </msubsup> </mrow> <mrow> <mo>|</mo> <mo>|</mo> <msubsup> <mi>p</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>-</mo> <msubsup> <mi>p</mi> <mi>l</mi> <mi>i</mi> </msubsup> <mo>|</mo> <msub> <mo>|</mo> <mn>2</mn> </msub> </mrow> </mfrac> </mrow> </mtd> </mtr> <mtr> <mtd> <mrow> <mo>+</mo> <mi>&beta;</mi> <munderover> <mi>&Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>L</mi> </munderover> <munderover> <mi>&Sigma;</mi> <mrow> <mi>k</mi> <mo>=</mo> <mn>1</mn> </mrow> <mrow> <msub> <mi>c</mi> <mn>1</mn> </msub> <mo>+</mo> <msub> <mi>c</mi> <mn>2</mn> </msub> </mrow> </munderover> <mo>|</mo> <mo>|</mo> <msubsup> <mi>w</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>|</mo> <msubsup> <mo>|</mo> <mn>2</mn> <mn>2</mn> </msubsup> <mo>)</mo> </mrow> </mtd> </mtr> </mtable> </mfenced>And then by all corresponding points received on sectioning imagePosition adjustment for optimization after positionI=1,2 ..., L, k=1,2 ..., c1+c2;Wherein,I is the sequence number of sequence section image;L is the sequence number of the current slice image;c1=N (i-1, i) represents the quantity of the corresponding points between the i-th -1 sectioning image and i-th of sectioning image;For the 1st A section, there is c1=0;c2=N (i, i+1) represents the quantity of the corresponding points between i-th of sectioning image and i+1 sectioning image;For working as Preceding l-th section, since the L+1 section does not obtain also, there is c2=0;K and l is the sequence number of corresponding points described in sequence section image, and k ≠ l;For the position coordinates of k-th of corresponding points in i-th of sectioning image;For the position coordinates of l-th of corresponding points in i-th of sectioning image;Respectively position coordinatesMotion vector;α and β is constant.
- 4. real-time registration apparatus according to claim 3, it is characterised in that using multi-grid method to min (E (w)) into Row rapid solving.
- 5. real-time registration apparatus according to claim 3, it is characterised in that Moving Least Squares are utilized on the GPU Method to carry out deformation to image.
- 6. real-time registration apparatus according to claim 5, it is characterised in that " often completed once to the corresponding points in CPU Adjustment after, the GPU all in accordance with the corresponding points after optimization position, to it is all received images carry out a shape Become, so as to complete to have received the once registering of image to all ", be specially:Different point v is chosen on the i-th width sectioning imagei, and corresponding rigid transformation matrix is calculated as follows<mrow> <mi>m</mi> <mi>i</mi> <mi>n</mi> <mrow> <mo>(</mo> <msub> <mi>l</mi> <msup> <mi>v</mi> <mi>i</mi> </msup> </msub> <mo>)</mo> </mrow> <mo>=</mo> <mi>m</mi> <mi>i</mi> <mi>n</mi> <mrow> <mo>(</mo> <munderover> <mi>&Sigma;</mi> <mrow> <mi>k</mi> <mo>=</mo> <mn>1</mn> </mrow> <mrow> <msub> <mi>c</mi> <mn>1</mn> </msub> <mo>+</mo> <msub> <mi>c</mi> <mn>2</mn> </msub> </mrow> </munderover> <msubsup> <mi>w</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>|</mo> <mo>|</mo> <msub> <mi>l</mi> <msup> <mi>v</mi> <mi>i</mi> </msup> </msub> <mo>(</mo> <msubsup> <mi>p</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>)</mo> <mo>-</mo> <msubsup> <mi>q</mi> <mi>k</mi> <mi>i</mi> </msubsup> <mo>|</mo> <msubsup> <mo>|</mo> <mn>2</mn> <mn>2</mn> </msubsup> <mo>)</mo> </mrow> </mrow>And then calculate the position coordinates after deformationUntil point all on the i-th width sectioning image completes deformation;Take i=1,2 ..., L, to it is all received images as stated above carry out a deformation so that complete to it is all Receive the once registering of image;Wherein,viFor the position at preceding any point of deformation in i-th of sectioning image;For viRigid transformation matrix to be calculated on point;For viPosition of the point after deformation;It is corresponding pointsPosition after adjustment;γ is constant;Represent corresponding pointsBy rigid transformationPosition coordinates afterwards.
- 7. real-time registration apparatus according to claim 6, it is characterised in that each difference of parallel computation on the GPU Point viCorresponding rigid transformation matrixWith the position after deformation
- 8. the real-time registration apparatus according to any one of claim 1-7, it is characterised in that the FPGA passes through kilomega network Network is connected with the scanning electron microscope and the calculation server respectively.
- 9. a kind of scanning electron microscope image Real-time Registration towards sequence section, it is characterised in that based in claim 1-8 Any one of them comprises the following steps towards the real-time registration apparatus of scanning electron microscope image of sequence section:Step S10, receives the slice image data from scanning electron microscope, and extract current slice image and a upper sectioning image it Between corresponding points;Step S20, according to all images having been received by and the corresponding points of extraction, to described in all received on image The position of corresponding points is adjusted, the position of the corresponding points after being optimized;All images that received according to the position of the corresponding points after optimization, are carried out deformation by step S30, so as to complete pair All registrations for having received image;Step S40, judges whether to have received last width sectioning image, otherwise, goes to step S10.
- 10. Real-time Registration according to claim 9, it is characterised in that current slice figure is extracted described in step S10 Picture and the corresponding points between a upper sectioning image, are specially:Calculate the SIFT feature of the current slice image, and the SIFT of the upper sectioning image with being buffered in FPGA Characteristic point is matched, and obtains the corresponding points between a upper sectioning image.
- 11. Real-time Registration according to claim 10, it is characterised in that after step slo, before step S20, Further include:Step S15, removes the SIFT feature for the upper sectioning image being buffered in FPGA, by the current slice image SIFT feature cached, for next sectioning image Feature Points Matching use.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201711248908.4A CN108038874B (en) | 2017-12-01 | 2017-12-01 | Scanning electron microscope image real-time registration device and method for sequence slices |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201711248908.4A CN108038874B (en) | 2017-12-01 | 2017-12-01 | Scanning electron microscope image real-time registration device and method for sequence slices |
Publications (2)
Publication Number | Publication Date |
---|---|
CN108038874A true CN108038874A (en) | 2018-05-15 |
CN108038874B CN108038874B (en) | 2020-07-24 |
Family
ID=62095075
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201711248908.4A Active CN108038874B (en) | 2017-12-01 | 2017-12-01 | Scanning electron microscope image real-time registration device and method for sequence slices |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN108038874B (en) |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109242894A (en) * | 2018-08-06 | 2019-01-18 | 广州视源电子科技股份有限公司 | Image alignment method and system based on mobile least square method |
CN110517242A (en) * | 2019-08-23 | 2019-11-29 | 强联智创(北京)科技有限公司 | A kind of aneurysmal analysis method and device |
CN112184638A (en) * | 2020-09-14 | 2021-01-05 | 南京市儿童医院 | Automatic kidney biopsy electron microscope picture identification method based on deep learning-comprehensive model |
CN112396608A (en) * | 2020-11-30 | 2021-02-23 | 中国科学院自动化研究所 | Biological tissue electron microscope image correction method, system and device based on X-ray image |
CN112988395A (en) * | 2021-04-20 | 2021-06-18 | 宁波兰茜生物科技有限公司 | Pathological analysis method and device of extensible heterogeneous edge computing framework |
CN114049252A (en) * | 2021-09-27 | 2022-02-15 | 中国科学院自动化研究所 | Scanning electron microscope three-dimensional image acquisition system and method for sequence slicing |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP4053729A4 (en) | 2020-09-23 | 2023-06-07 | Changxin Memory Technologies, Inc. | Chip product comparison method and apparatus, chip product modeling method and apparatus, and storage medium |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102920470A (en) * | 2012-10-18 | 2013-02-13 | 苏州生物医学工程技术研究所 | Medical imaging system and method with double-mode fusion |
US20130147789A1 (en) * | 2011-12-08 | 2013-06-13 | Electronics & Telecommunications Research Institute | Real-time three-dimensional real environment reconstruction apparatus and method |
CN103856727A (en) * | 2014-03-24 | 2014-06-11 | 北京工业大学 | Multichannel real-time video splicing processing system |
CN104616304A (en) * | 2015-02-11 | 2015-05-13 | 南京理工大学 | Self-adapting support weight stereo matching method based on field programmable gate array (FPGA) |
CN105279762A (en) * | 2015-11-20 | 2016-01-27 | 北京航空航天大学 | An oral cavity soft and hard tissue CT sequence and three-dimensional grid model registration method |
CN106708777A (en) * | 2017-01-23 | 2017-05-24 | 张军 | Multi-core heterogeneous CPU - CPU - FPGA architecture |
CN107247681A (en) * | 2017-05-27 | 2017-10-13 | 上海德衡数据科技有限公司 | A kind of big data engine prototype based on multinuclear isomery CPU GPU FPGA |
EP3249535A1 (en) * | 2016-05-25 | 2017-11-29 | Imagination Technologies Limited | Assessing performance of a hardware design using formal verification |
-
2017
- 2017-12-01 CN CN201711248908.4A patent/CN108038874B/en active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130147789A1 (en) * | 2011-12-08 | 2013-06-13 | Electronics & Telecommunications Research Institute | Real-time three-dimensional real environment reconstruction apparatus and method |
CN102920470A (en) * | 2012-10-18 | 2013-02-13 | 苏州生物医学工程技术研究所 | Medical imaging system and method with double-mode fusion |
CN103856727A (en) * | 2014-03-24 | 2014-06-11 | 北京工业大学 | Multichannel real-time video splicing processing system |
CN104616304A (en) * | 2015-02-11 | 2015-05-13 | 南京理工大学 | Self-adapting support weight stereo matching method based on field programmable gate array (FPGA) |
CN105279762A (en) * | 2015-11-20 | 2016-01-27 | 北京航空航天大学 | An oral cavity soft and hard tissue CT sequence and three-dimensional grid model registration method |
EP3249535A1 (en) * | 2016-05-25 | 2017-11-29 | Imagination Technologies Limited | Assessing performance of a hardware design using formal verification |
CN106708777A (en) * | 2017-01-23 | 2017-05-24 | 张军 | Multi-core heterogeneous CPU - CPU - FPGA architecture |
CN107247681A (en) * | 2017-05-27 | 2017-10-13 | 上海德衡数据科技有限公司 | A kind of big data engine prototype based on multinuclear isomery CPU GPU FPGA |
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109242894A (en) * | 2018-08-06 | 2019-01-18 | 广州视源电子科技股份有限公司 | Image alignment method and system based on mobile least square method |
CN109242894B (en) * | 2018-08-06 | 2021-04-09 | 广州视源电子科技股份有限公司 | Image alignment method and system based on mobile least square method |
CN110517242A (en) * | 2019-08-23 | 2019-11-29 | 强联智创(北京)科技有限公司 | A kind of aneurysmal analysis method and device |
CN112184638A (en) * | 2020-09-14 | 2021-01-05 | 南京市儿童医院 | Automatic kidney biopsy electron microscope picture identification method based on deep learning-comprehensive model |
CN112184638B (en) * | 2020-09-14 | 2024-02-06 | 南京市儿童医院 | Automatic identification method for kidney biopsy electron microscope picture based on deep learning-comprehensive model |
CN112396608A (en) * | 2020-11-30 | 2021-02-23 | 中国科学院自动化研究所 | Biological tissue electron microscope image correction method, system and device based on X-ray image |
CN112396608B (en) * | 2020-11-30 | 2021-05-04 | 中国科学院自动化研究所 | Biological tissue electron microscope image correction method, system and device based on X-ray image |
CN112988395A (en) * | 2021-04-20 | 2021-06-18 | 宁波兰茜生物科技有限公司 | Pathological analysis method and device of extensible heterogeneous edge computing framework |
CN112988395B (en) * | 2021-04-20 | 2021-08-24 | 宁波兰茜生物科技有限公司 | Pathological analysis method and device of extensible heterogeneous edge computing framework |
CN114049252A (en) * | 2021-09-27 | 2022-02-15 | 中国科学院自动化研究所 | Scanning electron microscope three-dimensional image acquisition system and method for sequence slicing |
Also Published As
Publication number | Publication date |
---|---|
CN108038874B (en) | 2020-07-24 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN108038874A (en) | Towards the real-time registration apparatus of scanning electron microscope image and method of sequence section | |
CN105447888B (en) | A kind of UAV Maneuver object detection method judged based on effective target | |
WO2011155150A1 (en) | Image processing apparatus, image processing method, and program | |
CN106600553A (en) | DEM super-resolution method based on convolutional neural network | |
CN108491757A (en) | Remote sensing image object detection method based on Analysis On Multi-scale Features study | |
CN103227888B (en) | A kind of based on empirical mode decomposition with the video stabilization method of multiple interpretational criteria | |
CN105744256A (en) | Three-dimensional image quality objective evaluation method based on graph-based visual saliency | |
CN106204447A (en) | The super resolution ratio reconstruction method with convolutional neural networks is divided based on total variance | |
CN106174830A (en) | Garment dimension automatic measurement system based on machine vision and measuring method thereof | |
CN107993250A (en) | A kind of fast multi-target pedestrian tracking and analysis method and its intelligent apparatus | |
CN109711401A (en) | A kind of Method for text detection in natural scene image based on Faster Rcnn | |
CN109000557A (en) | A kind of nuclear fuel rod pose automatic identifying method | |
CN106225681A (en) | A kind of Longspan Bridge health status monitoring device | |
CN107341844A (en) | A kind of real-time three-dimensional people's object plotting method based on more Kinect | |
DE112016001829T5 (en) | Automatically associate images using visual property references to related applications | |
CN112990077A (en) | Face action unit identification method and device based on joint learning and optical flow estimation | |
CN104463104B (en) | A kind of stationary vehicle target rapid detection method and device | |
CN107315994A (en) | Clustering algorithm based on Spectral Clustering space trackings | |
CN102663706A (en) | Adaptive weighted mean value filtering method based on diamond template | |
CN105335988B (en) | A kind of sub-pix center extraction method based on layered shaping | |
CN115760893A (en) | Single droplet particle size and speed measuring method based on nuclear correlation filtering algorithm | |
CN103914807B (en) | Non-locality image super-resolution method and system for zoom scale compensation | |
CN104463896B (en) | Image corner point detection method and system based on kernel similar region distribution characteristics | |
WO2022188030A1 (en) | Crowd density estimation method, electronic device and storage medium | |
CN106296741A (en) | Cell high-speed motion feature mask method in nanoscopic image |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
TR01 | Transfer of patent right | ||
TR01 | Transfer of patent right |
Effective date of registration: 20211110 Address after: 101407 room 413, floor 4, building 2, yard 11, Xingke East Street, Yanqi Economic Development Zone, Huairou District, Beijing Patentee after: Zhongke Guanwei (Beijing) Technology Co., Ltd Address before: 100190 No. 95, Zhongguancun East Road, Haidian District, Beijing Patentee before: Institute of automation, Chinese Academy of Sciences |