CN116912302A - High-precision imaging method and system based on depth image registration network - Google Patents
High-precision imaging method and system based on depth image registration network Download PDFInfo
- Publication number
- CN116912302A CN116912302A CN202311170392.1A CN202311170392A CN116912302A CN 116912302 A CN116912302 A CN 116912302A CN 202311170392 A CN202311170392 A CN 202311170392A CN 116912302 A CN116912302 A CN 116912302A
- Authority
- CN
- China
- Prior art keywords
- image
- axis
- imaging
- module
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 184
- 230000009466 transformation Effects 0.000 claims abstract description 103
- 239000011159 matrix material Substances 0.000 claims abstract description 69
- 230000036961 partial effect Effects 0.000 claims abstract description 69
- 238000000034 method Methods 0.000 claims abstract description 54
- 238000012545 processing Methods 0.000 claims abstract description 21
- 230000008569 process Effects 0.000 claims description 32
- 239000013598 vector Substances 0.000 claims description 29
- 238000000605 extraction Methods 0.000 claims description 17
- 230000001681 protective effect Effects 0.000 claims description 10
- 230000004927 fusion Effects 0.000 claims description 9
- 238000013507 mapping Methods 0.000 claims description 8
- 238000007500 overflow downdraw method Methods 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 6
- 230000011218 segmentation Effects 0.000 claims description 5
- 238000012549 training Methods 0.000 claims description 5
- 230000001131 transforming effect Effects 0.000 claims description 4
- 239000000047 product Substances 0.000 description 74
- 238000004519 manufacturing process Methods 0.000 description 6
- 238000010606 normalization Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- XEEYBQQBJWHFJM-UHFFFAOYSA-N Iron Chemical compound [Fe] XEEYBQQBJWHFJM-UHFFFAOYSA-N 0.000 description 2
- 230000004913 activation Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000005855 radiation Effects 0.000 description 2
- 229910000838 Al alloy Inorganic materials 0.000 description 1
- 206010073306 Exposure to radiation Diseases 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000002860 competitive effect Effects 0.000 description 1
- 239000012467 final product Substances 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000005865 ionizing radiation Effects 0.000 description 1
- 229910052742 iron Inorganic materials 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 229910052751 metal Inorganic materials 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/38—Registration of image sequences
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/30—Measuring arrangements characterised by the use of optical techniques for measuring roughness or irregularity of surfaces
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/30—Measuring arrangements characterised by the use of optical techniques for measuring roughness or irregularity of surfaces
- G01B11/306—Measuring arrangements characterised by the use of optical techniques for measuring roughness or irregularity of surfaces for measuring evenness
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/01—Arrangements or apparatus for facilitating the optical investigation
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/01—Arrangements or apparatus for facilitating the optical investigation
- G01N2021/0106—General arrangement of respective parts
- G01N2021/0112—Apparatus in one mechanical, optical or electronic block
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Pathology (AREA)
- Immunology (AREA)
- Biochemistry (AREA)
- Analytical Chemistry (AREA)
- Chemical & Material Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Artificial Intelligence (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a high-precision imaging method and a system based on a depth image registration network, wherein a high-precision imaging system is built, the system comprises a triaxial motion platform, an imaging module and a carrier, the carrier and the imaging module are respectively fixedly arranged on a Y axis and a Z axis of the triaxial motion platform, and a target product is fixed through the carrier; determining a shooting path of an imaging module by adjusting X, Y and a Z axis, and carrying out local shooting on a target product to obtain a plurality of local images; processing all adjacent images in the partial images by adopting a depth image registration network to obtain a transformation matrix of each group of adjacent images; and converting a plurality of partial images through a transformation matrix, sequentially filling the converted partial images into a blank large image designed in advance, and fusing adjacent images in the blank large image to obtain a high-precision complete image of the target product. The method can save the characteristic registration time of two adjacent partial images and has low system cost.
Description
Technical Field
The invention relates to the technical field of industrial vision, in particular to a high-precision imaging method and system based on a depth image registration network.
Background
In today's competitive manufacturing environment, enterprises need to continuously improve product quality, reduce costs, and maintain production efficiency and reliability. The application of high precision imaging techniques provides important support for achieving these goals. The high-precision imaging technology can provide accurate images and data and help industrial enterprises monitor the manufacturing process in real time. By means of the high-resolution imaging equipment, key links on the production line can be accurately recorded and detected, so that products can be ensured to meet quality standards. For example, in the production and processing of electronic and precision mechanical parts, high precision imaging can provide detailed images of the surface quality of objects for evaluating features such as surface finish, flatness, imperfections, and texture, thereby ensuring quality and consistency of each part and assembly.
At present, high precision imaging techniques typically require expensive equipment and complex image processing algorithms, which increase the cost of the application. Most implementations use X-ray equipment for imaging, however X-ray imaging uses ionizing radiation, long or excessive exposure may pose potential health risks to the human body and the environment, and secondly high precision imaging requires higher image resolution and quality, which may require increased radiation dose, which may increase radiation exposure risk. Therefore, there is a need to ensure effective control and monitoring of the radiation dose during imaging. The images produced by X-ray imaging techniques are typically gray scale images and may be difficult for a non-professional to interpret. Second, some high-precision imaging techniques at the present stage may require longer time to acquire and process image data, which may not be suitable for some application scenarios requiring real-time decision-making and feedback. For example, in the case where problems need to be found and corrected in time on a production line, a fast image processing speed is important.
The invention aims to provide a high-precision imaging method and a system based on a depth image registration network, which are used for realizing rapid high-precision imaging of a product to be detected. The depth image registration network can carry out rapid registration on the shot high-precision image, so that the calculated amount brought by the conventional registration method of the high-resolution image is reduced, the high-precision imaging speed is improved, and the robustness of the registration process by an algorithm is improved.
Disclosure of Invention
Aiming at the problems of high cost, low imaging speed and low imaging quality of the imaging equipment, the invention provides a high-precision imaging method and a high-precision imaging system based on a depth image registration network, which are used for realizing rapid high-precision imaging of a product to be detected.
In one aspect, the invention provides a high-precision imaging method based on a depth image registration network, which comprises the following steps:
s1, building a high-precision imaging system, wherein the system comprises a triaxial moving platform, an imaging module and a carrier, the carrier is fixedly arranged on a Y-axis of the triaxial moving platform, the imaging module is fixedly arranged on a Z-axis of the triaxial moving platform, and a target product is fixed through the carrier;
s2, the imaging module can clearly image the target product by adjusting the X axis, the Y axis and the Z axis until the imaging module can clearly image the target product, a shooting path of the imaging module is determined on the basis that the target product can be clearly imaged, and the imaging module locally shoots the target product according to the shooting path, so that a plurality of local image sequences with overlapping areas are obtained;
S3, presetting a depth image registration network and training to obtain a trained depth image registration network, and processing a plurality of local image sequences with overlapping areas by adopting the trained depth image registration network to obtain a plurality of transformation matrixes;
s4, converting a plurality of partial image sequences with overlapping areas through a plurality of transformation matrixes to obtain a plurality of converted partial images under the same coordinates, sequentially filling the plurality of converted partial images into a blank large image designed in advance, and fusing the overlapping areas of adjacent images in the blank large image by using a weighted fusion method to obtain a high-precision complete image of a target product.
Preferably, in S2, the imaging module may clearly image the target product by adjusting the X-axis, the Y-axis and the Z-axis until the imaging module determines a shooting path of the imaging module based on the clearly imaging of the target product, which specifically includes:
s21, enabling a target product to appear in the field of view of the imaging module by adjusting an X axis and a Y axis in the triaxial motion platform;
s22, enabling the imaging module to clearly image the target product by adjusting a Z axis in the triaxial motion platform, and recording the height of the Z axis at the moment;
S23, under the condition of Z-axis height determination, adjusting an X axis and a Y axis in a triaxial motion platform, determining a shooting starting point and an ending point of an imaging module, and recording a starting point coordinateAnd endpoint coordinates->;
S24, calibrating the imaging module by using a grid calibration method, and calculating the actual physical size corresponding to the photographed local image with the overlapping area;
s25, passing the starting point coordinatesEndpoint coordinates->And the actual physical size, and planning a shooting path of the imaging module.
Preferably, S25 specifically includes:
s251, passing the start point coordinatesAnd endpoint coordinates->Calculating the two-dimensional length of the region to be shot;
s252, dividing the two-dimensional length through the actual physical size, and determining the number of points to be shot;
s253, according to the starting point coordinatesCalculating the points to be shot to obtain coordinates of the intermediate process points;
s254, origin coordinatesIntermediate process point coordinates and end point coordinates +.>A photographing path of the imaging module is constituted.
Preferably, the coordinates of the intermediate process point in S253 are calculated as follows:
,
wherein ,,
,
,
,
in the formula ,representing the origin coordinates>Indicating endpoint coordinates +.>Representing intermediate process point coordinates, +.>Indicate->Go (go)/(go)>,/>Indicate->Column (S)/(S)>,/>Representation->The number of intermediate process points on the shaft, +. >Representation->The number of intermediate process points on the axis, length, represents the actual physical size corresponding to the partial image taken, units represents the length of the Pixels in a grid, and Pixels represents the actual length of a grid.
Preferably, in S3, a depth image registration network is preset and trained, where the depth image registration network includes a feature extraction module, a correlation estimation transform module, and a direct linear transformation module that are sequentially connected, where the feature extraction module is used to extract correlation features between two images in an image pair, the correlation estimation transform module is used to segment and linearly map the correlation features, estimate an offset between the two images in the image pair, and the direct linear transformation module converts the offset into a transformation matrix of the image pair.
Preferably, in S3, the trained depth image registration network is used to process a plurality of local image sequences with overlapping areas to obtain a plurality of transformation matrices, which specifically includes:
s31, preprocessing and grouping a plurality of local image sequences with overlapping areas to obtain a plurality of image pairs;
s32, sequentially selecting one image pair from a plurality of image pairs, and inputting the image pairs into a trained depth image registration network;
S33, respectively extracting features of the two images in the selected image pair by a feature extraction module to obtain vector features and mask features corresponding to each image, multiplying the vector features and the mask features corresponding to each image to obtain a feature matrix of each image overlapping area, and splicing the feature matrices of the two images in the selected image pair according to dimensions to obtain related features of the selected image pair;
S34, correlation estimation transducer module pairs correlation featuresPerforming segmentation processing and linear mapping to obtain initial characteristicsA sequence, which is used for processing the initial characteristic sequence and outputting an offset vector of the selected image pair;
s35, the direct linear transformation module calculates a transformation matrix of the selected image pair according to the offset vector;
s36, sequentially selecting another image pair from the plurality of image pairs until the plurality of image pairs are all selected, and obtaining a transformation matrix of the plurality of image pairs through processing in steps S32 to S35.
Preferably, the initial feature sequence in S34 can be formulated as:
;
in the formula ,representing an initial feature sequence>Weights representing the linear mapping of the image embedding operation procedure, +.>Representing a learnable one-dimensional position embedding vector, Representing a learnable class embedding vector, +.>Indicate->A sequence of the individual feature maps is shown,。
preferably, S4 specifically includes:
s41, dividing a transformation matrix of a plurality of image pairs into a plurality of column transformation matrices and a plurality of row transformation matrices, and calculating transformation matrices of other images in a plurality of partial image sequences with overlapping areas relative to the first image according to the plurality of column transformation matrices and the plurality of row transformation matrices;
s42, transforming other images in the local image sequences with the overlapping areas onto a main coordinate system by taking the coordinate system of the first image in the local image sequences with the overlapping areas as the main coordinate system through transformation matrixes corresponding to the first image respectively to obtain a transformed local image sequence;
s43, generating a blank large image by taking the upper left corner of the first image in a plurality of partial image sequences with overlapping areas as a starting point and the lower right corner of the last image in the transformed partial image sequences as an ending point;
s44, sequentially filling the first image and the transformed partial image sequence in the partial image sequences with the overlapping areas into the blank large image;
S45, carrying out linear weighted fusion on the overlapping areas of the adjacent images in the blank large image to obtain a high-precision complete image of the target product.
Preferably, in S41, the transformation matrix of the other images in the local image sequence with the overlapping area relative to the first image is calculated according to the column transformation matrices and the row transformation matrices, and the transformation matrix can be expressed as:
;
;
in the formula ,indicate->Line->Column image->Relative to the first image->Is a transformation matrix of->Indicating the first column is at the +.>Line and->Column transformation matrix of two adjacent images of a row, < >>Indicate->Line is at->Column and->Row transformation matrix of two adjacent images of a column, < >>Representing +.>Line->Image of column->For the post-conversion->Line->An image of a column.
In another aspect, the present invention provides a high-precision imaging system based on a depth image registration network, where the imaging is performed by using the high-precision imaging method based on the depth image registration network, where the high-precision imaging system includes: the high-precision imaging device is connected with the computer system, the computer system is provided with a depth image registration network, the high-precision imaging device comprises a base, a protective cover, a triaxial motion platform, an imaging module, a carrier and a control panel, the protective cover is arranged above the base, a semi-enclosed space is formed by surrounding the protective cover and the base, the triaxial motion platform is fixedly arranged on the base and located in the semi-enclosed space, the carrier is fixedly arranged on a Y-axis of the triaxial motion platform, the imaging module is fixedly arranged on a Z-axis of the triaxial motion platform, the control panel is arranged on one side of the base and close to the carrier, and a target product to be shot is located on the carrier, wherein:
The control panel is used for controlling the triaxial movement platform to move along X, Y and Z axes;
the three-axis motion platform drives the imaging module arranged on the three-axis motion platform to make relative motion with the target product so as to acquire a plurality of shooting positions;
the imaging module shoots a target product at a plurality of shooting positions to obtain a plurality of local image sequences with overlapping areas;
the computer system acquires a plurality of local image sequences with overlapping areas and processes the local image sequences through a depth image registration network arranged on the local image sequences, and outputs high-precision complete images of a target product.
The high-precision imaging method and the system based on the depth image registration network comprise the steps that firstly, a high-precision imaging system is built, the imaging system comprises a triaxial moving platform, an imaging module and a carrier, the carrier is fixedly arranged on a Y-axis of the triaxial moving platform, the imaging module is fixedly arranged on a Z-axis of the triaxial moving platform, and a target product is fixed through the carrier; then, the imaging module can clearly image the target product by adjusting the X axis, the Y axis and the Z axis of the triaxial moving platform, a shooting path of the imaging module is determined on the basis, and the imaging module locally shoots the target product according to the shooting path, so that a plurality of local image sequences with overlapping areas are obtained; presetting a depth image registration network and training to obtain a trained depth image registration network, and processing a plurality of local image sequences with overlapping areas by adopting the trained depth image registration network to obtain a plurality of transformation matrixes; and finally, converting a plurality of partial image sequences with overlapping areas through a transformation matrix to obtain a plurality of converted partial images under the same coordinates, sequentially filling the plurality of converted partial images into a blank large image designed in advance, and fusing adjacent images in the blank large image by using a weighted fusion method to obtain a high-precision complete image of the target product. The imaging module in the method comprises a high-precision camera, a telecentric lens and triaxial motion platform equipment, and compared with the existing imaging system, the equipment cost is greatly reduced; in addition, the depth image registration network in the method can accelerate the imaging speed by reducing the image resolution under the condition of unchanged imaging quality, and the time of feature registration is saved by estimating the transformation matrix of the image pair formed by adjacent images instead of relying on the traditional mode of detecting the feature points first and then registering.
Drawings
FIG. 1 is a flow chart of a high-precision imaging method based on a depth image registration network in an embodiment of the invention;
FIG. 2 is a photograph path of an imaging module according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a network architecture of a depth image registration network in accordance with an embodiment of the present invention;
FIG. 4 is a high-precision complete image contrast diagram after fusion of a partial image with an overlapping area of a target product and the target product according to an embodiment of the present invention, where (a) the partial image with the overlapping area of the target product is obtained by shooting with an imaging module, and (b) the high-precision complete image after fusion of the target product is obtained;
FIG. 5 is a schematic diagram of a system architecture of a high-precision imaging system according to an embodiment of the invention.
Reference numerals illustrate:
1. a base; 2. a protective cover; 3. a triaxial motion platform; 4. an imaging module; 5. a carrier; 6. a control panel;
41. a high-precision camera; 42. a telecentric lens; 43. a polychromatic light source.
Detailed Description
In order to make the technical scheme of the present invention better understood by those skilled in the art, the present invention will be further described in detail with reference to the accompanying drawings.
A high-precision imaging method based on a depth image registration network specifically comprises the following steps:
S1, building a high-precision imaging system, wherein the system comprises a triaxial moving platform, an imaging module and a carrier, the carrier is fixedly arranged on a Y-axis of the triaxial moving platform, the imaging module is fixedly arranged on a Z-axis of the triaxial moving platform, and a target product is fixed through the carrier;
s2, the imaging module can clearly image the target product by adjusting the X axis, the Y axis and the Z axis until the imaging module can clearly image the target product, a shooting path of the imaging module is determined on the basis that the target product can be clearly imaged, and the imaging module locally shoots the target product according to the shooting path, so that a plurality of local image sequences with overlapping areas are obtained;
s3, presetting a depth image registration network and training to obtain a trained depth image registration network, and processing a plurality of local image sequences with overlapping areas by adopting the trained depth image registration network to obtain a plurality of transformation matrixes;
s4, converting a plurality of partial image sequences with overlapping areas through a plurality of transformation matrixes to obtain a plurality of converted partial images under the same coordinates, sequentially filling the plurality of converted partial images into a blank large image designed in advance, and fusing the overlapping areas of adjacent images in the blank large image by using a weighted fusion method to obtain a high-precision complete image of a target product.
Specifically, referring to fig. 1, fig. 1 is a flowchart of a high-precision imaging method based on a depth image registration network according to an embodiment of the present invention.
Firstly, a high-precision imaging system is built, the imaging system comprises a triaxial moving platform, an imaging module and a carrier, wherein the carrier is fixedly arranged on a Y axis of the triaxial moving platform, the imaging module is fixedly arranged on a Z axis of the triaxial moving platform, a target product is fixed through the carrier, the imaging module can clearly image the target product by adjusting an X axis, a Y axis and a Z axis, on the basis, a shooting path of the imaging module is determined, the imaging module locally shoots the target product according to the shooting path, and a plurality of local image sequences with overlapping areas of the target product are obtained;
the imaging system further comprises a computer, a depth image registration network is preset on the computer and trained to obtain a trained depth image registration network, a plurality of local image sequences with overlapping areas are preprocessed and grouped to obtain a plurality of image pairs, and each image pair is processed by adopting the trained depth image registration network to obtain a transformation matrix of each image pair; converting a plurality of partial images with overlapping areas shot by an imaging module through a corresponding transformation matrix to obtain a plurality of converted partial images with overlapping areas under the same coordinate, sequentially filling the converted partial images with overlapping areas into a blank large image designed in advance, and fusing adjacent images in the blank large image by using a weighted fusion method to obtain a high-precision complete image of a target product.
In one embodiment, in S2, the imaging module may clearly image the target product by adjusting the X-axis, the Y-axis and the Z-axis until the imaging module determines a shooting path of the imaging module based on the clearly imaging of the target product, which specifically includes:
s21, enabling a target product to appear in the field of view of the imaging module by adjusting an X axis and a Y axis in the triaxial motion platform;
s22, enabling the imaging module to clearly image the target product by adjusting a Z axis in the triaxial motion platform, and recording the height of the Z axis at the moment;
s23, under the condition of Z-axis height determination, adjusting an X axis and a Y axis in a triaxial motion platform, determining a shooting starting point and an ending point of an imaging module, and recording a starting point coordinateAnd endpointCoordinates->;
S24, calibrating the imaging module by using a grid calibration method, and calculating the actual physical size corresponding to the photographed local image with the overlapping area;
s25, passing the starting point coordinatesEndpoint coordinates->And the actual physical size, and planning a shooting path of the imaging module.
In one embodiment, S25 specifically includes:
s251, passing the start point coordinatesAnd endpoint coordinates->Calculating the two-dimensional length of the region to be shot;
s252, dividing the two-dimensional length through the actual physical size, and determining the number of points to be shot;
S253, according to the starting point coordinatesCalculating the points to be shot to obtain coordinates of the intermediate process points;
s254, origin coordinatesIntermediate process point coordinates and end point coordinates +.>A photographing path of the imaging module is constituted.
In one embodiment, the coordinates of the intermediate process point in S253 are calculated as follows:
,
wherein ,,
,
,
,
in the formula ,representing the origin coordinates>Indicating endpoint coordinates +.>Representing intermediate process point coordinates, +.>Indicate->Go (go)/(go)>,/>Indicate->Column (S)/(S)>,/>Representation->The number of intermediate process points on the shaft, +.>Representation->The number of intermediate process points on the axis, length, represents the actual physical size corresponding to the partial image taken, units represents the length of the Pixels in a grid, and Pixels represents the actual length of a grid.
Specifically, referring to fig. 2, fig. 2 is a schematic view of a photographing path of an imaging module according to an embodiment of the present invention.
Planning a shooting path of an imaging module comprises the following steps:
1) Adjusting the X axis and the Y axis of the triaxial moving platform, and moving the target product into the field of view of the imaging module, so that an image of the target product exists in the field of view of the imaging module in the focusing process;
2) Adjusting the Z-axis height of the triaxial moving platform, enabling an imaging module fixed on the Z-axis to clearly image a target product, and recording the Z-axis height at the moment;
3) Under the condition of determining the Z-axis height, the X-axis and the Y-axis of the triaxial moving platform are adjusted again, the shooting starting point and the shooting end point of the imaging module when the target product is locally shot are determined, and the starting point coordinates are recordedAnd endpoint coordinates;
4) The imaging module is calibrated by using a grid calibration method, specifically, the length Units (unit: number of Pixels) and the actual length Pixels of one grid (unit: mm), the actual physical size corresponding to the photographed partial image is obtained through the length of the pixels in one grid and the actual length of one grid:
,
wherein Length is the actual physical size corresponding to the local image, units is the length of Pixels in a grid, pixels is the actual length of a grid, 5120 is the resolution of the high-precision camera in the imaging module, and also corresponds to the resolution of the local image with the overlapping region.
5) From the coordinates of the starting pointAnd endpoint coordinates->The two-dimensional length (unit: mm) of the region to be photographed is calculated, and the specific formula is as follows:
,
6) Calculating the points required to be shot by the imaging module in the X axis and the Y axis respectively:
,
wherein ,representing a downward rounding;
7) According to the coordinates of the starting point And calculating coordinates of the intermediate process points according to the points required to be shot:
,
wherein ,indicate->Go (go)/(go)>,/>Indicate->Column (S)/(S)>Thus obtaining->Coordinates of the intermediate shooting points.
8) And planning a moving route according to the starting point coordinates, the intermediate process point coordinates and the end point coordinates to obtain a shooting path of the imaging module shown in fig. 2. The imaging module sequentially acquires the local area images of the target product at each shooting point according to the planned shooting path to obtain the target productA high precision partial image sequence with overlapping regions is provided.
In one embodiment, in S3, a depth image registration network is preset and trained, where the depth image registration network includes a feature extraction module, a correlation estimation transform module, and a direct linear transformation module that are sequentially connected, where the feature extraction module is used to extract correlation features between two images in an image pair, the correlation estimation transform module is used to segment and linearly map the correlation features, estimate an offset between the two images in the image pair, and the direct linear transformation module converts the offset into a transformation matrix of the image pair.
In one embodiment, in S3, the trained depth image registration network is used to process a plurality of local image sequences with overlapping areas to obtain a plurality of transformation matrices, which specifically includes:
S31, preprocessing and grouping a plurality of local image sequences with overlapping areas to obtain a plurality of image pairs;
s32, sequentially selecting one image pair from a plurality of image pairs, and inputting the image pairs into a trained depth image registration network;
s33, respectively extracting features of the two images in the selected image pair by a feature extraction module to obtain vector features and mask features corresponding to each image, multiplying the vector features and the mask features corresponding to each image to obtain a feature matrix of each image overlapping area, and splicing the feature matrices of the two images in the selected image pair according to dimensions to obtain related features of the selected image pair;
S34, correlation estimation transducer module pairs correlation featuresPerforming segmentation processing and linear mapping to obtain an initial feature sequence, processing the initial feature sequence, and outputting an offset vector of the selected image pair;
s35, the direct linear transformation module calculates a transformation matrix of the selected image pair according to the offset vector;
s36, sequentially selecting another image pair from the plurality of image pairs until the plurality of image pairs are all selected, and obtaining a transformation matrix of the plurality of image pairs through processing in steps S32 to S35.
In one embodiment, the initial signature sequence in S34 may be formulated as:
;
in the formula ,representing an initial feature sequence>Weights representing the linear mapping of the image embedding operation procedure, +.>Representing a learnable one-dimensional position embedding vector,representing a learnable class embedding vector, +.>Indicate->A sequence of the individual feature maps is shown,。
specifically, referring to fig. 3, fig. 3 is a network structure schematic diagram of the depth image registration network.
The depth image registration network in fig. 3 includes a feature extraction module, a correlation estimation transform module and a direct linear transformation module, which are sequentially connected, where the feature extraction module is used to extract correlation features between two images in an image pair, the correlation estimation transform module is used to segment and linearly map the correlation features, estimate offset between the two images in the image pair, and the direct linear transformation module converts the offset into a transformation matrix of the image pair.
Processing a plurality of local image sequences with overlapping areas by adopting a depth image registration network to obtain a plurality of transformation matrixes, wherein the specific process is as follows:
1) Obtained by shootingPreprocessing a partial image sequence with an overlapping region, specifically, firstly adjusting the resolution of each partial image with the overlapping region to a preset size, for example 512X512, so as to reduce the calculated amount of a depth image registration network; each partial image with the resolution adjusted overlap region is then normalized, i.e. using the empirical mean [118.93, 113.97, 102.60 ] ]And empirical variances [69.85, 68.81, 72.45]Normalizing the pixel values of the partial image from 0 to 255 to 0 to 1, respectively, thereby obtaining +.>A sequence of pre-processed partial images,
for a pair ofThe partial image sequences after preprocessing are grouped according to the line, and the same group of +.>Two adjacent images in the partial image after pretreatment are divided into a group to form +.>Pairs of images; the first preprocessed partial images of each line are also adjacently combined to form +.>Image pairs. Through the operation, the method canThe partial image sequence after the preprocessing is converted into +.>Image pairs.
2) From theEach image pair is sequentially selected from each image pair and is input to a feature extraction module, the feature extraction module comprises a feature extraction submodule and a mask generation submodule which are arranged in parallel, wherein the feature extraction submodule is a 4-layer convolution module, the convolution module consists of a convolution layer, a batch normalization layer and an activation function layer, and the sizes of the feature extraction submodules are all [1,512,512 ]]Vector features of> and />The method comprises the steps of carrying out a first treatment on the surface of the The mask generation sub-module is a 5-layer volumeThe convolution module is also composed of a convolution layer, a batch normalization layer and an activation function layer, and the size of the convolution module is [1,512,512 ] ]And a value of 0 or 1 vector mask +.> and />Wherein a position of matrix 0 indicates that the feature of the position is invalid, a position of matrix 1 indicates that the feature of the position is valid, and a vector mask is used to mask out the feature of the non-overlapping region; multiplying the characteristic matrix of each image in the input image by the mask matrix to obtain the characteristic matrix of each image: and />Finally, the characteristic matrix of the two images in the input image pair is +.>Andstitching by dimension to form [2,512,512 ] of the input image pair]Relevant features of the dimension matrix->。
3) Will [2,512,512 ]]Correlation features of a dimension matrixThe method comprises the steps of inputting a correlation estimation transducer module, wherein the correlation estimation transducer module comprises a feature segmentation sub-module and a transducer encoder sub-module, and the feature segmentation sub-module receives correlation features->And dividing it into a sequence of feature maps of a predetermined number, predetermined size, for example +.>8 pieces of [2,64,64 ] size]Is->The feature map sequence is firstly linearly mapped and then overlapped with position information to form an initial feature sequence, and the initial feature sequence can be expressed as follows by a formula:
;
in the formula ,representing an initial feature sequence>Weights representing the linear mapping of the image embedding operation procedure, +. >Representing a learnable one-dimensional position embedding vector,representing a learnable class embedding vector, +.>Indicate->A sequence of the individual feature maps is shown,。
4) The initial characteristic sequence is carried outInput to a transducer encoder sub-module composed of L transducer modules connected in series, each transducer module including a first layer normalization module, a Multi-Head Attention Module (MSA), and a first layer normalization module connected in sequenceAnd the two-layer normalization module and the feedforward network module. Wherein the multi-head Attention module enhances the Attention mechanism by adding a plurality of different subspaces such that each subspace can focus on a different subset of information, each multi-head Attention Module (MSA) comprises a plurality of Self-Attention modules (SA, self-Attention) consisting of a plurality of Self-Attention modules which are grouped by aggregating an initial feature sequence>To extract relevant features.
Specifically, for the firstCharacteristic sequence input by the individual transducer module +.>Each element of which is associated with a weight matrix of three different learnable parameters +.>、/>、/>Multiplying to obtain the query matrix of the current module>Key matrix->Sum matrix->:
;
;
;
wherein ,is->Query matrix of the individual transducer modules, < > >Is->Key matrix of the individual transducer modules>,/>Is->Value matrix of the individual transducer modules, < >>Is->Characteristic sequences output by the individual transducer modules, < >>、/>、/>Is->A weight matrix of three different learnable parameters of a transducer module.
Then pass through the firstQuery matrix of the individual transducer modules>And key matrix->Dot product operation between, calculate +.>Attention weight of the individual transducer module +.>:
;
wherein ,is the attention weight.
Then weight attentionAND value matrix->Dot product operation is performed to obtain a value of self-attention module +.>:
;
The multi-head attention Module (MSA) repeats the self-attention module (SA) 4 times above, concatenating the context vectors of each head output, and then projects them back into an 8-dimensional context vector by linear transformationThe specific formula can be expressed as:
;
wherein ,a weight matrix representing a linear transformation in the multi-headed attention module,indicate->The output matrix of the ith self-attention module of the transducer module.
The transducer encoder sub-module concatenates L multi-headed attention Modules (MSAs) using a residual structure and inputs the calculated vectors to a feed forward network to estimate an offset vector of 8 offsets of 4 vertices of the two images of the input image pair The formula can be expressed as:
;
;
;
wherein ,representation layer normalization operation, ++>Representing the use of a multi-layer perceptron to output classification results, < + >>Indicating that go through->Results of multiple head attention Module (MSA), -, for example>Indicate->Output result of individual Tansformer module, < >>Representing the number of multi-head attention Modules (MSA) in series, the last multi-head attention Module (MSA) in series outputting the resultRepresenting the predicted offset vector.
5) The offset vector is processed by a Direct Linear Transformation (DLT) module to calculate a transformation matrix of the input image pair。
In one embodiment, S4 specifically includes:
s41, dividing a transformation matrix of a plurality of image pairs into a plurality of column transformation matrices and a plurality of row transformation matrices, and calculating transformation matrices of other images in a plurality of partial image sequences with overlapping areas relative to the first image according to the plurality of column transformation matrices and the plurality of row transformation matrices;
s42, transforming other images in the local image sequences with the overlapping areas onto a main coordinate system by taking the coordinate system of the first image in the local image sequences with the overlapping areas as the main coordinate system through transformation matrixes corresponding to the first image respectively to obtain a transformed local image sequence;
S43, generating a blank large image by taking the upper left corner of the first image in a plurality of partial image sequences with overlapping areas as a starting point and the lower right corner of the last image in the transformed partial image sequences as an ending point;
s44, sequentially filling the first image and the transformed partial image sequence in the partial image sequences with the overlapping areas into the blank large image;
s45, carrying out linear weighted fusion on the overlapping areas of the adjacent images in the blank large image to obtain a high-precision complete image of the target product.
In one embodiment, in S41, a transformation matrix of the other images in the local image sequence with the overlapping area relative to the first image is calculated according to a plurality of column transformation matrices and a plurality of row transformation matrices, where the transformation matrix can be expressed as:
;
;
in the formula ,indicate->Line->Column image->Relative to the first image->Is a transformation matrix of->Indicating the first column is at the +.>Line and->Column transformation matrix of two adjacent images of a row, < >>Indicate->Line is at->Column and->Row transformation matrix of two adjacent images of a column, < >>Representing +.>Line->Image of column- >For the post-conversion->Line->An image of a column.
Specifically, the method for calculating the high-precision complete image of the target product according to the transformation matrix of the plurality of image pairs comprises the following steps:
1) Will beThe transformation matrix of the individual image pairs is divided into +.>Individual row transform matrices and->A column transform matrix, wherein:
the individual row transformation matrices are respectively:
;
the individual column transformation matrices are respectively:
。
the above-mentioned "row transformation matrix" means a transformation matrix of images of two adjacent columns for a certain row, for exampleA transformation matrix representing first and second column images of a first row; the above-mentioned "column transformation matrix" means a transformation matrix for images of two adjacent rows of the first column, e.g., +.>A transformation matrix representing a first row image and a second row image of a first column.
2) The transformation matrix of other images and the first image in the partial images with the overlapped area is calculated according to the row transformation matrix and the column transformation matrix, and the specific formula is as follows:
;
;
in the formula ,?>Line->Column image->Relative to the first image->Is a transformation matrix of->Is->Line->The column of images are transformed into an aligned image. />
And transforming other images (images except the first image) in the partial images with the overlapping areas onto the main coordinate system through the transformation matrix of the images and the first image by taking the coordinate system of the first partial image in the partial image sequence with the overlapping areas shot by the imaging module as the main coordinate system, so as to obtain a transformed partial image sequence. After the transformation, the local image sequence with the overlapping area shot by the imaging module takes the first image as a reference and is aligned in the coordinate system of the first image.
3) With a first image of a plurality of partial images with overlapping areasThe upper left corner of (a) as the starting pointWith the last image in the transformed partial image sequence +.>Generating a blank large image by taking the right lower corner of the first image as the end point, and sequentially adding +.>And the transformed partial image sequence +.>Filling the images into the blank large image, and using weighted fusion to the overlapping area of the adjacent images on the blank large image to eliminate the problem of inaccurate registration. The weighted fusion can be formulated as follows:
;
wherein , and />Representing the area sequentially filled in two adjacent images in the blank large image, < >>Andrespectively representing the first and the second image of two adjacent images in the main coordinate system +.>The pixel value of the lower one of the pixel values, and />Respectively representing the adjacent two images in the main coordinate system +.>The weight of the fusion region below. After linear weighting fusion is carried out on the overlapped areas of all the adjacent images, the final product is obtainedTo a high precision complete image of the target product. Referring to fig. 4, fig. 4 is a comparison chart of a fused high-precision complete image of a target product and a local image of the target product with an overlapping area in an embodiment of the present invention, where (a) is a local image of the target product with an overlapping area obtained by shooting with an imaging module, and (b) is a fused high-precision complete image of the target product. And processing 35 partial images with overlapping areas of the target product obtained through shooting by adopting the method to obtain a high-precision complete image of the fused target product.
A high-precision imaging system based on a depth image registration network, imaging by adopting a high-precision imaging method based on the depth image registration network, the high-precision imaging system comprising: base, safety cover, triaxial motion platform, imaging module, carrier and control panel, the base top is located to the safety cover, and the safety cover surrounds with the base and forms semi-closed space, and triaxial motion platform sets firmly on the base and is located semi-closed space, and the carrier sets firmly on triaxial motion platform's the Y axle, and imaging module sets firmly on triaxial motion platform's the Z axle, and control panel sets up in the side of base and is close to carrier one side, and the target product that waits to shoot is located the carrier, wherein:
the control panel is used for controlling the triaxial movement platform to move along X, Y and Z axes;
the three-axis motion platform drives the imaging module arranged on the three-axis motion platform to make relative motion with the target product so as to acquire a plurality of shooting positions;
the imaging module shoots the target product at a plurality of shooting positions to obtain a plurality of partial images with overlapping areas.
The computer system acquires a plurality of partial images with overlapping areas and processes the partial images through a depth image registration network arranged on the partial images, and outputs a high-precision complete image of a target product.
Specifically, referring to fig. 5, fig. 5 is a schematic system configuration diagram of a high-precision imaging system according to an embodiment of the invention.
The high-precision imaging system comprises a high-precision imaging device and a computer system (not shown in fig. 5), wherein the high-precision imaging device is connected with the computer system, and a depth image registration network is arranged in the computer system.
The high-precision imaging device comprises a base 1, a protective cover 2, a triaxial moving platform 3, an imaging module 4, a carrier 5 and a control panel 6, wherein the protective cover 2 is arranged above the base 1, the protective cover 2 and the base 1 are encircled to form a semi-enclosed space, the triaxial moving platform 3 is fixedly arranged on the base 1 and located in the semi-enclosed space, the carrier 5 is fixedly arranged on a Y-axis of the triaxial moving platform, the imaging module 4 is fixedly arranged on a Z-axis of the triaxial moving platform, the control panel 6 is arranged on the side face of the base and close to one side of the carrier 5, and a target product to be photographed is located on the carrier 5.
The control panel 6 controls the triaxial moving platform 3 to drive the imaging module 4 arranged on the triaxial moving platform and the target product to perform relative movement so as to acquire a plurality of shooting positions; the imaging module 4 shoots a target product at a plurality of shooting positions to obtain a plurality of partial images with overlapping areas; the computer system acquires a plurality of partial images with overlapping areas and processes the partial images through a depth image registration network arranged on the partial images, and outputs a high-precision complete image of a target product.
Further, the imaging module 4 includes a high-precision camera 41, a telecentric lens 42, a polychromatic light source 43: the high-precision camera 41 is a 2500-ten-thousand-pixel tera-net-area-array RGB camera, has a resolution of 5120x5120, and outputs an RGB three-channel image at a maximum frame rate of 41.5 FPS; telecentric lens 42 is a double telecentric lens, has optical distortion of 0.023%, magnification of 0.634, depth of field of 12.5mm, resolution of 6.8um, and object field of 36.3mm, and can clearly observe local area of product without distortion; the multicolor light source 43 is an AOI light source, has RGB three-color light sources, and can uniformly irradiate the product to reduce the influence of uneven brightness.
Further, the high-precision electromagnetic motion modules are used for the X axis and the Y axis of the triaxial motion platform 3, so that the accuracy of a target product in moving shooting can be guaranteed, and the lead screw motion module is used for the Z axis of the triaxial motion platform 3, so that the shake of a camera module can be reduced.
Further, the base 1 is an integrally formed aluminum alloy base and is used for fixing the triaxial moving platform 3 so as to reduce shake of each axis of the triaxial moving platform 3 in the moving process and improve imaging precision.
Further, the protection cover 2 is an iron plate bent through a metal plate, and a space formed by the protection cover 2 and the base 1 in a surrounding mode is used for an imaging device and a triaxial motion platform arranged in the protection cover 2.
The high-precision imaging method and the system based on the depth image registration network comprise the steps that firstly, a high-precision imaging system is built, the imaging system comprises a triaxial moving platform, an imaging module and a carrier, the carrier is fixedly arranged on a Y-axis of the triaxial moving platform, the imaging module is fixedly arranged on a Z-axis of the triaxial moving platform, and a target product is fixed through the carrier; then, the imaging module can clearly image the target product by adjusting the X axis, the Y axis and the Z axis of the triaxial moving platform, a shooting path of the imaging module is determined on the basis, and the imaging module locally shoots the target product according to the shooting path, so that a plurality of local image sequences with overlapping areas are obtained; presetting a depth image registration network and training to obtain a trained depth image registration network, and processing a plurality of local image sequences with overlapping areas by adopting the trained depth image registration network to obtain a plurality of transformation matrixes; and finally, converting a plurality of partial image sequences with overlapping areas through a transformation matrix to obtain a plurality of converted partial images under the same coordinates, sequentially filling the plurality of converted partial images into a blank large image designed in advance, and fusing adjacent images in the blank large image by using a weighted fusion method to obtain a high-precision complete image of the target product. The imaging module in the method comprises a high-precision camera, a telecentric lens and triaxial motion platform equipment, and compared with the existing imaging system, the equipment cost is greatly reduced; in addition, the depth image registration network in the method can accelerate the imaging speed by reducing the image resolution under the condition of unchanged imaging quality, and the time of feature registration is saved by estimating the transformation matrix of the image pair formed by adjacent images instead of relying on the traditional mode of detecting the feature points first and then registering.
The high-precision imaging method and the system based on the depth image registration network provided by the invention are described in detail. The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to facilitate an understanding of the core concepts of the invention. It should be noted that it will be apparent to those skilled in the art that various modifications and adaptations of the invention can be made without departing from the principles of the invention and these modifications and adaptations are intended to be within the scope of the invention as defined in the following claims.
Claims (10)
1. A depth image registration network-based high-precision imaging method, the method comprising:
s1, building a high-precision imaging system, wherein the system comprises a triaxial moving platform, an imaging module and a carrier, the carrier is fixedly arranged on a Y-axis of the triaxial moving platform, the imaging module is fixedly arranged on a Z-axis of the triaxial moving platform, and a target product is fixed through the carrier;
s2, the imaging module can clearly image the target product by adjusting an X axis, a Y axis and a Z axis until the imaging module can clearly image the target product, a shooting path of the imaging module is determined on the basis of clearly imaging the target product, and the imaging module locally shoots the target product according to the shooting path, so that a plurality of local image sequences with overlapping areas are obtained;
S3, presetting a depth image registration network and training to obtain a trained depth image registration network, and processing a plurality of local image sequences with overlapping areas by adopting the trained depth image registration network to obtain a plurality of transformation matrixes;
s4, converting the partial image sequences with the overlapping areas through the transformation matrixes to obtain converted partial images under the same coordinates, sequentially filling the converted partial images into blank large images designed in advance, and fusing the overlapping areas of adjacent images in the blank large images by using a weighted fusion method to obtain the high-precision complete image of the target product.
2. The high-precision imaging method based on the depth image registration network according to claim 1, wherein the step of determining the shooting path of the imaging module based on the clear imaging of the target product by adjusting the X-axis, the Y-axis and the Z-axis until the imaging module can clearly image the target product in the step of S2 specifically comprises:
s21, enabling the target product to appear in the field of view of the imaging module by adjusting an X axis and a Y axis in the triaxial motion platform;
S22, enabling the imaging module to clearly image the target product by adjusting a Z axis in the triaxial motion platform, and recording the height of the Z axis at the moment;
s23, under the condition of Z-axis height determination, adjusting an X axis and a Y axis in the triaxial motion platform, determining a shooting starting point and an ending point of the imaging module, and recording starting point coordinatesAnd endpoint coordinates->;
S24, calibrating the imaging module by using a grid calibration method, and calculating the actual physical size corresponding to the photographed local image with the overlapping area;
s25, passing through the starting point coordinatesEndpoint coordinates->And the actual physical size is used for planning a shooting path of the imaging module.
3. The high-precision imaging method based on the depth image registration network according to claim 2, wherein S25 specifically comprises:
s251, passing stationThe origin coordinatesAnd endpoint coordinates->Calculating the two-dimensional length of the region to be shot;
s252, dividing the two-dimensional length according to the actual physical size, and determining the number of points to be shot;
s253, according to the starting point coordinatesCalculating the points to be shot to obtain coordinates of intermediate process points;
s254, the starting point coordinates Intermediate process point coordinates and end point coordinates +.>A photographing path of the imaging module is constituted.
4. A depth image registration network based high precision imaging method according to claim 3, wherein the coordinates of the intermediate process point in S253 are calculated as follows:
,
wherein ,,
,
,
,
in the formula ,representing the origin coordinates>Indicating endpoint coordinates +.>Representing intermediate process point coordinates, +.>Indicate->Go (go)/(go)>,/>Indicate->Column (S)/(S)>,/>Representation->The number of intermediate process points on the shaft, +.>Representation of/>The number of intermediate process points on the axis, length, represents the actual physical size corresponding to the partial image taken, units represents the length of the Pixels in a grid, and Pixels represents the actual length of a grid.
5. The high-precision imaging method based on a depth image registration network according to claim 4, wherein the depth image registration network is preset and trained in S3, the depth image registration network comprises a feature extraction module, a correlation estimation transform module and a direct linear transformation module which are sequentially connected, the feature extraction module is used for extracting correlation features between two images in an image pair, the correlation estimation transform module is used for dividing and linearly mapping the correlation features, estimating offset between the two images in the image pair, and the direct linear transformation module is used for converting the offset into a transformation matrix of the image pair.
6. The high-precision imaging method based on a depth image registration network according to claim 5, wherein in S3, the trained depth image registration network is adopted to process a plurality of local image sequences with overlapping areas to obtain a plurality of transformation matrices, and the method specifically comprises:
s31, preprocessing and grouping a plurality of local image sequences with overlapping areas to obtain a plurality of image pairs;
s32, sequentially selecting one image pair from a plurality of image pairs and inputting the image pairs into the trained depth image registration network;
s33, the feature extraction module performs feature extraction on two images in the selected image pair respectively to obtain vector features and mask features corresponding to each image, multiplies the vector features and the mask features corresponding to each image to obtain feature matrixes of each image overlapping area, and performs dimension processing on the feature matrixes of the two images in the selected image pairLine stitching to obtain the relevant features of the selected image pairs;
S34, the correlation estimation transducer module pairs the correlation characteristicsPerforming segmentation processing and linear mapping to obtain an initial feature sequence, processing the initial feature sequence, and outputting an offset vector of a selected image pair;
S35, the direct linear transformation module calculates a transformation matrix of the selected image pair according to the offset vector;
s36, sequentially selecting another image pair from the plurality of image pairs until the plurality of image pairs are all selected, and obtaining a transformation matrix of the plurality of image pairs through processing in steps S32 to S35.
7. The depth image registration network-based high precision imaging method of claim 6, wherein the initial feature sequence in S34 is formulated as:
;
in the formula ,representing an initial feature sequence>Weights representing the linear mapping of the image embedding operation procedure, +.>Representing a learnable one-dimensional position embedding vector, +.>Representing a learnable class embedding vector, +.>Indicate->Sequence of feature maps, ">。
8. The depth image registration network-based high-precision imaging method of claim 7, wherein S4 specifically comprises:
s41, dividing a transformation matrix of a plurality of image pairs into a plurality of column transformation matrices and a plurality of row transformation matrices, and calculating transformation matrices of other images in the local image sequences with the overlapping areas relative to the first image according to the plurality of column transformation matrices and the plurality of row transformation matrices;
S42, transforming other images in the local image sequences with the overlapping areas onto a main coordinate system by taking the coordinate system of the first image in the local image sequences with the overlapping areas as the main coordinate system through transformation matrixes corresponding to the first image respectively to obtain a transformed local image sequence;
s43, generating a blank large image by taking the upper left corner of the first image in the partial image sequence with the overlapping area as a starting point and the lower right corner of the last image in the partial image sequence after transformation as an end point;
s44, sequentially filling the first image in the partial image sequences with the overlapped areas and the transformed partial image sequences into the blank large image;
s45, performing linear weighted fusion on the overlapping areas of the adjacent images in the blank large image to obtain a high-precision complete image of the target product.
9. The method of high precision imaging based on a depth image registration network according to claim 8, wherein in S41, a transformation matrix of the other images in the local image sequence with overlapping regions relative to the first image is calculated according to a number of column transformation matrices and a number of row transformation matrices, and the transformation matrix can be expressed as:
;
;
in the formula ,indicate->Line->Column image->Relative to the first image->Is a transformation matrix of->Indicating the first column is at the +.>Line and->Column transformation matrix of two adjacent images of a row, < >>Indicate->Line is at->Column and->Row transformation matrix of two adjacent images of a column, < >>Representing +.>Line->The image of the column is displayed,for the post-conversion->Line->An image of a column.
10. A depth image registration network-based high-precision imaging system for imaging using the depth image registration network-based high-precision imaging method as recited in any one of claims 1 to 9, the high-precision imaging system comprising: the high-precision imaging device is connected with the computer system, a depth image registration network is arranged in the computer system, the high-precision imaging device comprises a base, a protective cover, a triaxial moving platform, an imaging module, a carrier and a control panel, the protective cover is arranged above the base, the protective cover and the base surround to form a semi-closed space, the triaxial moving platform is fixedly arranged on the base and positioned in the semi-closed space, the carrier is fixedly arranged on a Y-axis of the triaxial moving platform, the imaging module is fixedly arranged on a Z-axis of the triaxial moving platform, the control panel is arranged on the side face of the base and is close to one side of the carrier, and a target product to be photographed is positioned on the carrier, wherein:
The control panel is used for controlling the triaxial motion platform to move along X, Y and Z axes;
the three-axis motion platform drives the imaging module arranged on the three-axis motion platform to perform relative motion with the target product so as to acquire a plurality of shooting positions;
the imaging module shoots the target product at a plurality of shooting positions to obtain a plurality of local image sequences with overlapping areas;
and the computer system acquires a plurality of local image sequences with overlapping areas, processes the local image sequences through a depth image registration network arranged on the local image sequences, and outputs a high-precision complete image of the target product.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311170392.1A CN116912302B (en) | 2023-09-12 | 2023-09-12 | High-precision imaging method and system based on depth image registration network |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311170392.1A CN116912302B (en) | 2023-09-12 | 2023-09-12 | High-precision imaging method and system based on depth image registration network |
Publications (2)
Publication Number | Publication Date |
---|---|
CN116912302A true CN116912302A (en) | 2023-10-20 |
CN116912302B CN116912302B (en) | 2023-12-01 |
Family
ID=88351490
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202311170392.1A Active CN116912302B (en) | 2023-09-12 | 2023-09-12 | High-precision imaging method and system based on depth image registration network |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN116912302B (en) |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2011174799A (en) * | 2010-02-24 | 2011-09-08 | Mitsubishi Electric Corp | Photographing route calculation device |
US20150221116A1 (en) * | 2013-05-29 | 2015-08-06 | Capso Vision, Inc. | Method of Overlap-Dependent Image Stitching for Images Captured Using a Capsule Camera |
CN107146201A (en) * | 2017-05-08 | 2017-09-08 | 重庆邮电大学 | A kind of image split-joint method based on improvement image co-registration |
CN113273172A (en) * | 2020-08-12 | 2021-08-17 | 深圳市大疆创新科技有限公司 | Panorama shooting method, device and system and computer readable storage medium |
US20220012850A1 (en) * | 2018-11-15 | 2022-01-13 | The Regents Of The University Of California | System and method for transforming holographic microscopy images to microscopy images of various modalities |
WO2022205625A1 (en) * | 2021-03-30 | 2022-10-06 | 广东拓斯达科技股份有限公司 | Dispensing path generation method and apparatus, electronic device, and storage medium |
WO2022242395A1 (en) * | 2021-05-20 | 2022-11-24 | 北京城市网邻信息技术有限公司 | Image processing method and apparatus, electronic device and computer-readable storage medium |
CN115526781A (en) * | 2022-10-12 | 2022-12-27 | 中国人民解放军陆军工程大学 | Splicing method, system, equipment and medium based on image overlapping area |
EP4160140A1 (en) * | 2021-10-04 | 2023-04-05 | VoxelSensors SRL | Three dimensional imaging system |
CN116017123A (en) * | 2022-12-05 | 2023-04-25 | 武汉华中天经通视科技有限公司 | Photoelectric imaging system and imaging method thereof |
-
2023
- 2023-09-12 CN CN202311170392.1A patent/CN116912302B/en active Active
Patent Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2011174799A (en) * | 2010-02-24 | 2011-09-08 | Mitsubishi Electric Corp | Photographing route calculation device |
US20150221116A1 (en) * | 2013-05-29 | 2015-08-06 | Capso Vision, Inc. | Method of Overlap-Dependent Image Stitching for Images Captured Using a Capsule Camera |
CN107146201A (en) * | 2017-05-08 | 2017-09-08 | 重庆邮电大学 | A kind of image split-joint method based on improvement image co-registration |
US20220012850A1 (en) * | 2018-11-15 | 2022-01-13 | The Regents Of The University Of California | System and method for transforming holographic microscopy images to microscopy images of various modalities |
CN113273172A (en) * | 2020-08-12 | 2021-08-17 | 深圳市大疆创新科技有限公司 | Panorama shooting method, device and system and computer readable storage medium |
WO2022205625A1 (en) * | 2021-03-30 | 2022-10-06 | 广东拓斯达科技股份有限公司 | Dispensing path generation method and apparatus, electronic device, and storage medium |
WO2022242395A1 (en) * | 2021-05-20 | 2022-11-24 | 北京城市网邻信息技术有限公司 | Image processing method and apparatus, electronic device and computer-readable storage medium |
EP4160140A1 (en) * | 2021-10-04 | 2023-04-05 | VoxelSensors SRL | Three dimensional imaging system |
CN115526781A (en) * | 2022-10-12 | 2022-12-27 | 中国人民解放军陆军工程大学 | Splicing method, system, equipment and medium based on image overlapping area |
CN116017123A (en) * | 2022-12-05 | 2023-04-25 | 武汉华中天经通视科技有限公司 | Photoelectric imaging system and imaging method thereof |
Non-Patent Citations (2)
Title |
---|
PAN WU等: "Multi-focus image fusion: Transformer and shallow feature attention matters", 《DISPLAYS》, pages 1 - 10 * |
张超等: "叶型孔图像拼接技术实验研究", 《航空精密制造技术》, pages 13 - 16 * |
Also Published As
Publication number | Publication date |
---|---|
CN116912302B (en) | 2023-12-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
RU2601421C2 (en) | Method and system of calibrating camera | |
US9269131B2 (en) | Image processing apparatus with function of geometrically deforming image, image processing method therefor, and storage medium | |
US20230370577A1 (en) | Calibration method and apparatus for binocular camera, image correction method and apparatus for binocular camera, storage medium, terminal and intelligent device | |
JPH0818848A (en) | Camera system and distortion removing method | |
US11488279B2 (en) | Image processing apparatus, image processing system, imaging apparatus, image processing method, and storage medium | |
US20050100245A1 (en) | Method for correcting distortions in multi-focus image stacks | |
JP2021165944A (en) | Learning method, program, and image processing apparatus | |
CN110853102B (en) | Novel robot vision calibration and guide method and device and computer equipment | |
CN111986267A (en) | Coordinate system calibration method of multi-camera vision system | |
CN116912302B (en) | High-precision imaging method and system based on depth image registration network | |
TW201740311A (en) | Facial image-processing method and system thereof | |
CN109064511B (en) | Method and device for measuring height of center of gravity of human body and related equipment | |
CN116276938B (en) | Mechanical arm positioning error compensation method and device based on multi-zero visual guidance | |
CN112381888A (en) | Dynamic compensation method for H-shaped steel cutting path | |
JP7204586B2 (en) | LEARNING METHOD, PROGRAM AND IMAGE PROCESSING DEVICE | |
CN111507904A (en) | Image splicing method and device for microcosmic printed patterns | |
CN113960111B (en) | Three-dimensional thermal imaging system and method based on thermal imager and laser linkage scanning technology | |
JP2020030569A (en) | Image processing method, image processing device, imaging device, lens device, program, and storage medium | |
CN116095488A (en) | Optical lens driving method and device and electronic equipment | |
US20230143670A1 (en) | Automated Image Acquisition System for Automated Training of Artificial Intelligence Algorithms to Recognize Objects and Their Position and Orientation | |
CN214200141U (en) | Robot repeated positioning precision measuring system based on vision | |
JP2011118767A (en) | Facial expression monitoring method and facial expression monitoring apparatus | |
CN117812466B (en) | Calibration method, device and system for large-scene double-camera linkage camera | |
CN111508071B (en) | Binocular camera-based 3D modeling method and shooting terminal | |
RU2789190C1 (en) | Underwater video camera calibration method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |