CN116611994A

CN116611994A - Image stitching method and device, and glass through hole image positioning method and device

Info

Publication number: CN116611994A
Application number: CN202211475233.8A
Authority: CN
Inventors: 袁佳汝; 周朝阳
Original assignee: Dier Laser Technology Wuxi Co ltd
Current assignee: Dier Laser Technology Wuxi Co ltd
Priority date: 2022-11-23
Filing date: 2022-11-23
Publication date: 2023-08-18

Abstract

The invention provides an image stitching method, an image stitching device, a glass through hole image positioning method and equipment, wherein the image stitching method comprises the following steps: acquiring an image to be stitched, wherein the image to be stitched comprises a first local image and a second local image, the first local image comprises a first overlapping area overlapped with the second local image, and the second local image comprises a second overlapping area overlapped with the first local image; determining a matching area in the first overlapping area, and determining a first round hole feature matrix of a round hole array in the matching area and a second round hole feature matrix of a round hole array in the second overlapping area; traversing a second round hole feature matrix based on the first round hole feature matrix, and determining a feature similarity matrix with highest similarity with the first round hole feature matrix in the second round hole feature matrix; and splicing the first partial image and the second partial image based on the first round hole feature matrix and the feature similarity matrix. The invention uses the round hole characteristics to splice the images, thereby improving the image splicing efficiency and accuracy.

Description

Image stitching method and device, and glass through hole image positioning method and device

Technical Field

The invention belongs to the field of digital image processing, and particularly relates to an image stitching method, an image stitching device, a glass through hole image positioning method and a glass through hole image positioning device.

Background

In order for an electronic device to achieve its preset function, an array of circular holes needs to be provided in its structural members. As electronic devices are light and thin and miniaturized, the size of round holes in structural members in the electronic devices is also becoming smaller and smaller, and the development is proceeding toward micropores with diameters of tens of micrometers. In order to ensure the stability and reliability of the performance of the electronic equipment, all round holes need to be photographed for round hole quality detection. In order to improve the precision of the circular hole image, the shooting field of view of the camera needs to be reduced, namely: the breadth of photographing every time is reduced, the complete structural part cannot be displayed in one image, so that a plurality of partial images with circular hole arrays are needed to be spliced, the accurate positions of defective points and the positions of defective products in the complete structural part are conveniently positioned, and subsequent processing is facilitated.

For the glass through hole product, a plurality of round holes with extremely small size (tens of micrometers) and in row-column arrangement are processed in glass, the number of the round holes is large, the sizes are close, the shapes are similar, the surface of the glass through hole product is smooth, the color of the glass is single, and the conventional image splicing method cannot accurately and efficiently finish splicing.

For example, the following several ways of stitching images are known in the art: 1. image stitching is achieved based on a scale invariant feature transform matching algorithm (Scale Invariant Feature Transform, SIFT) algorithm; 2. image stitching is realized based on the existing template matching algorithm; 3. and the high-precision motor is used for driving the structural member to move, a plurality of partial images are obtained at a plurality of preset shooting positions, and the partial images are directly spliced based on motor parameters. The three prior arts have the following technical problems when applied to image splicing of glass through hole products with circular hole arrays: 1. the first method has the advantages that the calculation amount is extremely large, real-time calculation cannot be performed, and because the spliced objects are a plurality of circular hole arrays with extremely small apertures, the characteristics of the circular holes are very close, the proper characteristics are difficult to extract by the characteristic change matching algorithm and the conventional characteristic extraction algorithm, and the same coincident circular hole cannot be accurately found; 2. the second prior art is generally pixel matching, the calculated amount is large when the algorithm is directly adopted, the splicing efficiency is low, the round holes of the round hole array are almost identical, and when the pixels are singly matched, the images are spliced in a staggered manner or are matched in a staggered manner, so that the splicing error is caused; 3. the third type has extremely high requirements on motor precision, and the conventional motor can not meet the micrometer-level requirements at present due to communication delay in the photographing process, so that the splicing error can be caused.

Disclosure of Invention

In view of the foregoing, it is necessary to provide an image stitching method, an image stitching device, a glass through hole image positioning method and a glass through hole image positioning device, so as to solve the technical problems in the prior art that stitching errors are likely to occur when stitching images with similar features or stitching images with similar features is caused by relying on higher motor precision, and the stitching efficiency is low.

In one aspect, the present invention provides an image stitching method for stitching images to be stitched having a circular hole array, the image stitching method comprising:

acquiring an image to be spliced, wherein the image to be spliced comprises a first local image and a second local image, the first local image comprises a first overlapping area overlapped with the second local image, and the second local image comprises a second overlapping area overlapped with the first local image;

determining a matching region in the first overlapping region, and determining a first circular hole feature matrix of a circular hole array in the matching region and a second circular hole feature matrix of a circular hole array in the second overlapping region;

traversing the second round hole feature matrix based on the first round hole feature matrix, and determining a feature similarity matrix with highest similarity with the first round hole feature matrix in the second round hole feature matrix;

And splicing the first partial image and the second partial image based on the first round hole feature matrix and the feature similarity matrix.

In some possible implementations, the first round hole feature matrix includes at least one of a first round hole outer diameter matrix, a first round hole inner diameter matrix, a first round hole wall thickness matrix, a first round hole gray scale matrix, and a first round hole dirt matrix; the second round hole feature matrix comprises at least one of a second round hole outer diameter matrix, a second round hole inner diameter matrix, a second round hole wall thickness matrix, a second round hole gray scale matrix and a second round hole dirt matrix.

In some possible implementations, the determining the first circular hole feature matrix of the circular hole array in the matching area includes:

determining first round hole characteristics of each first round hole in the round hole array in the matching region;

determining first round hole coordinates of each first round hole in the round hole array in the matching region;

determining the first circular hole feature matrix based on the first circular hole feature and the first circular hole coordinates;

the determining a second circular hole matrix of the circular hole array in the second overlapping region includes:

determining a second hole feature for each second hole in the array of holes in the second overlapping region;

Determining second round hole coordinates of each second round hole in the round hole array in the second overlapping region;

and determining the second round hole feature matrix based on the second round hole feature and the second round hole coordinates.

In some possible implementations, the first round hole feature includes at least one of a first outer diameter, a first inner diameter, a first wall thickness, and the second round hole feature includes at least one of a second outer diameter, a second inner diameter, a second wall thickness;

the determining the first hole feature of each first hole in the array of holes in the matching region includes:

preprocessing the matching area to obtain the outer contour of each first round hole and the inner contour of each first round hole;

determining the first outer diameter based on the outer contour of the first circular hole, and/or determining the first inner diameter based on the inner contour of the first circular hole, and/or determining the first wall thickness based on the outer contour of the first circular hole and the inner contour of the first circular hole;

the determining a second hole feature for each second hole in the array of holes in the second overlap region comprises:

preprocessing the second overlapping area to obtain the outer contour of each second round hole and the inner contour of each second round hole;

The second outer diameter is determined based on the outer contour of the second circular hole, and/or the second inner diameter is determined based on the inner contour of the second circular hole, and/or the second wall thickness is determined based on the outer contour of the second circular hole and the inner contour of the second circular hole.

In some possible implementations, the first circular hole feature further includes a first average gray value and/or a first smudge value, and the second circular hole feature further includes a second average gray value and/or a second smudge value;

the determining the first hole feature of each first hole in the array of holes in the matching region further includes:

determining a first round hole area of the first round hole based on the outer contour of the first round hole or the inner contour of the first round hole, and determining a first average gray value of the first round hole area;

and/or the number of the groups of groups,

determining gray values of preset areas around the first round holes;

judging whether dirt exists in the preset area around the first round hole or not based on the gray value of the preset area around the first round hole;

when the gray value of the preset area around the first round hole is smaller than a gray threshold value, dirt exists in the preset area around the first round hole, and the dirt value of the first round hole is set to be a first value;

When the gray value of the preset area around the first round hole is larger than or equal to a gray threshold value, no dirt exists in the preset area around the first round hole, and the dirt value of the first round hole is set to be a second value;

the determining a second hole feature for each second hole in the array of holes in the second overlap region further comprises:

determining a second round hole area of the second round hole based on the outer contour of the second round hole or the inner contour of the second round hole, and determining a first average gray value of the second round hole area;

and/or the number of the groups of groups,

determining gray values of preset areas around the second round holes;

judging whether dirt exists in the peripheral preset area of the second round hole or not based on the gray value of the peripheral preset area of the second round hole;

when the gray value of the preset area around the second round hole is smaller than the gray threshold value, dirt exists in the preset area around the second round hole, and the dirt value of the second round hole is set to be a first value;

when the gray value of the preset area around the second round hole is larger than or equal to the gray threshold value, no dirt exists in the preset area around the second round hole, and the dirt value of the second round hole is set to be a second value.

In some possible implementations, the image to be stitched further includes a third partial image, and the image stitching method includes:

acquiring the first partial image in real time based on a shooting camera, acquiring the characteristics of all round holes in the first partial image, and determining the first round hole characteristic matrix based on the characteristics of all round holes in the first partial image and the matching area;

acquiring the second partial image in real time based on a shooting camera, acquiring the characteristics of all round holes in the second partial image, and determining the second round hole characteristic matrix based on the characteristics of all round holes in the second partial image and the second overlapping region;

splicing the first partial image and the second partial image based on the first round hole feature matrix and the second round hole feature matrix to obtain a partial spliced image, and simultaneously acquiring the third partial image based on a shooting camera in real time; wherein the second partial image includes a third overlapping region overlapping the third partial image, the third partial image including a fourth overlapping region overlapping the second partial image;

determining a reference area in the third overlapping area, determining a third round hole feature matrix of a round hole array in the reference area based on round hole features of all round holes in the second partial image and the reference area, and determining a fourth round hole feature matrix of the round hole array in the fourth overlapping area based on round hole features of all round holes in the third partial image and the fourth overlapping area;

And splicing the local spliced image and the third local image based on the third round hole feature matrix and the fourth round hole feature matrix to obtain a spliced image.

In some possible implementations, the second round hole feature matrix includes a plurality of second round hole feature sub-matrices; the determining the feature similarity matrix with the highest similarity with the first round hole feature matrix in the second round hole feature matrix comprises the following steps:

determining the inner diameter similarity, the outer diameter similarity, the wall thickness similarity, the gray level similarity and the dirt similarity of the first circular hole feature matrix and each second circular hole feature sub-matrix;

acquiring a first weight of the inner diameter similarity, a second weight of the outer diameter similarity, a third weight of the wall thickness similarity, a fourth weight of the gray scale similarity and a fifth weight of the dirt similarity;

determining the comprehensive similarity of the first round hole feature matrix and each of the second round hole feature sub-matrices based on the inner diameter similarity, the first weight, the outer diameter similarity, the second weight, the wall thickness similarity, the third weight, the gray scale similarity, the fourth weight, the dirt similarity and the fifth weight;

And taking the second round hole feature submatrix with the highest comprehensive similarity as the feature similarity matrix.

On the other hand, the invention also provides an image stitching device, which is used for stitching images to be stitched with a circular hole array, and comprises the following steps:

the image acquisition unit is used for acquiring an image to be spliced, wherein the image to be spliced comprises a first local image and a second local image, the first local image comprises a first overlapping area overlapped with the second local image, and the second local image comprises a second overlapping area overlapped with the first local image;

a feature matrix determining unit, configured to determine a matching region in the first overlapping region, and determine a first circular hole feature matrix of a circular hole array in the matching region and a second circular hole feature matrix of a circular hole array in the second overlapping region;

the characteristic similarity matrix determining unit is used for traversing the second round hole characteristic matrix based on the first round hole characteristic matrix and determining a characteristic similarity matrix with highest similarity with the first round hole characteristic matrix in the second round hole characteristic matrix;

and the image stitching unit is used for stitching the first partial image and the second partial image based on the first round hole feature matrix and the feature similarity matrix.

On the other hand, the invention also provides a glass through hole image positioning method, which is characterized by comprising the following steps:

shooting a glass through hole product, obtaining a plurality of glass through hole images, and splicing the glass through hole images based on an image splicing method to obtain spliced images;

positioning bad points in the spliced image;

the image stitching method is the image stitching method in any one of the possible implementation modes.

In some possible implementations, the pore size of each round hole in the glass via product is 20 μm to 100 μm.

On the other hand, the invention also provides an image splicing device which comprises a memory and a processor, wherein,

the memory is used for storing programs;

the processor is coupled to the memory, and is configured to execute the program stored in the memory, so as to implement the steps in the image stitching method in any one of the possible implementations.

The beneficial effects of adopting the embodiment are as follows: according to the image stitching method, firstly, a first circular hole feature matrix of a matching area and a second circular hole feature matrix of a second overlapping area are determined, then, a feature similarity matrix with highest feature matching similarity is determined in the second circular hole feature matrix according to the first circular hole feature matrix, and finally, a first local image and a second local image are stitched based on the first circular hole feature matrix and the feature similarity matrix. The feature used in the image splicing process is the round hole feature of each round hole, but not the pixel feature of each pixel point in the image, and the order of magnitude of the pixel points is far larger than that of the round holes, so that the calculation complexity can be reduced from the number of pixels to the number of round holes by using the round hole feature to splice the image, the calculation amount in the image splicing process is greatly reduced, and the image splicing efficiency is improved. Further, compared with the pixel characteristics, the circular hole characteristic matrix is used for splicing the images, so that splicing errors caused when different circular hole pixel characteristics are the same are avoided, and the accuracy of image splicing is improved.

Furthermore, after the graph splicing is completed, the positions of the defective points in the whole glass through hole product can be accurately positioned, and the accuracy of the positioning of the defective points on the spliced image can be improved by improving the accuracy of the image splicing, so that an operator or a mechanical arm can be guided to process the positioned defective points.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic flow chart of an embodiment of an image stitching method according to the present invention;

FIG. 2 is a schematic structural diagram of an embodiment of a first overlapping region, a second overlapping region, and a matching region according to the present invention;

FIG. 3 is a flowchart illustrating an embodiment of determining the first circular hole feature matrix in S102 of FIG. 1 according to the present invention;

FIG. 4 is a schematic diagram of an embodiment of the row and column positions of the circular holes in the first circular hole array, taking the matching area as an example, when the circular hole feature matrix is established;

FIG. 5 is a flowchart illustrating an embodiment of determining the second circular hole feature matrix in S102 of FIG. 1 according to the present invention;

FIG. 6 is a flowchart illustrating the process of S301 in FIG. 3 according to an embodiment of the present invention;

FIG. 7 is a flowchart illustrating the process of S501 in FIG. 5 according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of an embodiment of a single circular hole image provided by the present invention;

FIG. 9 is a schematic structural diagram of an embodiment of average gray values of a circular hole according to the present invention;

FIG. 10 is a schematic view of the structure of an embodiment of a round hole with and without dirt around the round hole provided by the invention;

FIG. 11 is a flowchart illustrating the process of S301 in FIG. 3 according to another embodiment of the present invention;

FIG. 12 is a flowchart illustrating the step S501 of FIG. 5 according to another embodiment of the present invention;

FIG. 13 is a schematic flow chart diagram of an embodiment of stitching multiple partial images according to the present invention;

FIG. 14 is a flowchart illustrating the step S103 of FIG. 1 according to the present invention;

FIG. 15 is a schematic structural diagram of an embodiment of an image stitching device according to the present invention;

fig. 16 is a schematic structural diagram of an embodiment of an image stitching device provided in the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

It should be understood that the schematic drawings are not drawn to scale. A flowchart, as used in this disclosure, illustrates operations implemented according to some embodiments of the present invention. It should be appreciated that the operations of the flow diagrams may be implemented out of order and that steps without logical context may be performed in reverse order or concurrently. Moreover, one or more other operations may be added to or removed from the flow diagrams by those skilled in the art under the direction of the present disclosure. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor systems and/or microcontroller systems.

References to "first," "second," etc. in the embodiments of the present invention are for descriptive purposes only and are not to be construed as indicating or implying a relative importance or the number of technical features indicated. Thus, a technical feature defining "first", "second" may include at least one such feature, either explicitly or implicitly. "and/or", describes an association relationship of an associated object, meaning that there may be three relationships, for example: a and/or B may represent: a exists alone, A and B exist together, and B exists alone.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the invention. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.

The invention provides an image stitching method, an image stitching device, a glass through hole image positioning method and glass through hole image positioning equipment, and the method and the device are respectively described below.

Fig. 1 is a schematic flow chart of an embodiment of an image stitching method provided by the present invention, where, as shown in fig. 1, the image stitching method includes:

s101, acquiring an image to be spliced, wherein the image to be spliced comprises a first local image and a second local image, the first local image comprises a first overlapping area overlapped with the second local image, and the second local image comprises a second overlapping area overlapped with the first local image;

s102, determining a matching area in the first overlapping area, and determining a first round hole feature matrix of a round hole array in the matching area and a second round hole feature matrix of a round hole array in the second overlapping area;

S103, traversing a second round hole feature matrix based on the first round hole feature matrix, and determining a feature similarity matrix with highest similarity with the first round hole feature matrix in the second round hole feature matrix;

s104, splicing the first partial image and the second partial image based on the first round hole feature matrix and the feature similarity matrix.

Compared with the prior art, the image stitching method provided by the embodiment of the invention comprises the steps of firstly determining a first circular hole feature matrix of a matching area and a second circular hole feature matrix of a second overlapping area, then determining a feature similarity matrix with highest similarity with the first circular hole feature matrix in the second circular hole feature matrix, and finally stitching the first partial image and the second partial image based on the first circular hole feature matrix and the feature similarity matrix. The feature used in the image splicing process is the round hole feature of each round hole, but not the pixel feature of each pixel point in the image, and the order of magnitude of the pixel points is far larger than that of the round holes, so that the embodiment of the invention can reduce the operation complexity from the number of pixels to the number of round holes by using the round hole feature to splice the image, thereby greatly reducing the calculated amount in the image splicing process and improving the image splicing efficiency. Further, compared with the pixel characteristics, the circular hole characteristic matrix is used for splicing the images, so that splicing errors caused when different circular hole pixel characteristics are the same are avoided, and the accuracy of image splicing is improved.

Furthermore, the embodiment of the invention only needs to ensure that the first partial image and the second partial image have overlapping areas, has no strict requirement on the acquisition positions of the first partial image and the second partial image, reduces the acquisition precision of the first partial image and the second partial image, realizes the aim of correct splicing even when the acquisition precision of the first partial image and the second partial image is lower, further improves the accuracy of image splicing, and simultaneously reduces the hardware cost of an image acquisition device for acquiring the first partial image and the second partial image.

Furthermore, after pattern splicing is completed, the positions of the defective points in the whole glass through hole product can be accurately positioned, and the accuracy of the positioning of the defective points on the spliced images can be improved by improving the accuracy of image splicing, so that operators or mechanical arms can be guided to carry out subsequent processing on the positioned defective points. The bad points are that the round hole machining quality possibly occurring after the glass through hole product is machined does not meet the machining requirement, such as the round hole is oversized or undersized, the round hole is not machined through or blocked in the machining process, or dirty spots appear around the round hole.

It should be noted that: in the embodiment of the invention, the circular hole array is taken as an example for description, if some glass through holes are required to be square or triangular due to the requirements of process products, the images to be spliced can also be images with square hole arrays and triangular hole arrays.

It should be understood that: the first partial image and the second partial image may be obtained by capturing the structural member in real time based on the capturing system in the step S101, or may be obtained by calling from a storage medium storing the first partial image and the second partial image.

In a specific embodiment of the invention, the image to be stitched is an image with an array of glass vias (TGV).

In the embodiment of the invention, the first partial image and the second partial image are spliced through the round hole features with smaller orders of magnitude, so in the preferred embodiment of the invention, the first partial image and the second partial image are obtained by shooting the structural member in real time based on the shooting system in the mode of obtaining the first partial image and the second partial image in step S101, and after the first partial image and the second partial image are obtained by shooting in real time, the first partial image and the second partial image are directly spliced, so that the instantaneity of image splicing is improved.

It should be noted that: the size of the first overlap region and the second overlap region may be obtained from a field of view of a camera in the imaging system and a step length of the structure at each imaging. Specifically: for example, when the camera view is 12mm and the step length of the structure is 10mm, the area width of the first overlapping area and the second overlapping area is 2mm.

Also to be described is: because the motor driving the structural member to displace has a motion error and communication delay exists in the shooting exposure process, the area widths of the first overlapping area and the second overlapping area are not completely the same, so in order to ensure the similarity of the first circular hole feature matrix and the second circular hole feature matrix, in some embodiments of the invention, the matching area should be located at the center position of the first overlapping area as much as possible, namely: the center of the matching region should coincide as much as possible with the center of the first overlap region, in particular: the difference between the center of the matching region and the center of the first overlap region should be less than or equal to a preset difference.

The preset difference may be set or adjusted according to the size of the matching area and the size of the first overlapping area, which is not specifically limited herein.

In a specific embodiment of the present invention, the first overlapping region, the second overlapping region, and the matching region are as shown in fig. 2.

In some embodiments of the invention, the first round hole feature matrix comprises at least one of a first round hole outer diameter matrix, a first round hole inner diameter matrix, a first round hole wall thickness matrix, a first round hole gray scale matrix, and a first round hole dirt matrix; the second round hole feature matrix comprises at least one of a second round hole outer diameter matrix, a second round hole inner diameter matrix, a second round hole wall thickness matrix, a second round hole gray scale matrix and a second round hole dirt matrix.

According to the embodiment of the invention, the first round hole feature matrix and the second round hole feature matrix are arranged to comprise the feature matrices of the physical features of the round holes of different types, so that the diversity of image splicing modes can be improved. And when the first round hole feature matrix and the second round hole feature matrix are arranged to comprise feature matrices of various round hole physical features, images can be spliced from the various round hole physical features, and the accuracy of image splicing is further improved.

It should be noted that: when the image to be spliced is an image with a square hole array and a triangular hole array, the first round hole feature matrix may be a first square hole feature matrix, where the first square hole feature matrix includes at least one of a first square Kong Bianchang matrix, a first square hole diagonal matrix, a first square hole wall thickness matrix, a first square Kong Huidu matrix, and a first square Kong Zangwu matrix. And will not be described in detail herein.

In some embodiments of the present invention, as shown in fig. 3, determining the first circular hole feature matrix in step S102 includes:

s301, determining first round hole characteristics of each first round hole in the round hole array in the matching area;

s302, determining first round hole coordinates of each first round hole in the round hole array in the matching area;

s303, determining a first round hole feature matrix based on the first round hole feature and the first round hole coordinates.

According to the embodiment of the invention, the first round hole feature matrix comprises the first round hole coordinates, and when images are spliced, the reliability of round hole matching can be determined through the first round hole coordinates, so that the technical problem of wrong rows or wrong columns in the image splicing process is further avoided, and the accuracy of image splicing is further improved.

It should be understood that, since the quality detection is performed on the spliced image after the splicing is performed on the spliced image, that is, the defective point is located, the quality detection is determined by the first circular hole feature, and the quality detection is performed not only on the matching area, the method of determining the first circular hole feature according to the embodiment of the present invention may be as follows: the first round hole characteristics of all round holes in the round hole array in the first partial image are determined, the first round hole characteristics of the matching area are determined from the first round hole characteristics of all round holes, the round hole characteristics of the first partial image can be determined once, and the quality detection efficiency of the image is improved.

In a specific embodiment of the present invention, as shown in fig. 4, the first round hole coordinates are two-dimensional coordinates, which are used to represent the row and column where the round holes are located, respectively, and the first round hole of the first row and column is denoted as (1, 1).

In some embodiments of the present invention, as shown in fig. 5, determining the second circular hole feature matrix in step S102 includes:

s501, determining second round hole characteristics of each second round hole in the round hole array in the second overlapping region;

s502, determining second round hole coordinates of each second round hole in the round hole array in the second overlapping region;

s503, determining a second round hole feature matrix based on the second round hole feature and the second round hole coordinates.

It should be noted that: the second circular hole coordinates are two-dimensional coordinates, and the representation method is the same as that of the first circular hole coordinates, and is not repeated here.

Also to be described is: the specific manner of determining the first circular hole coordinates in step S302 and the second circular hole coordinates in step S502 may be: determined by a line number determination function (findnzero).

In some embodiments of the invention, the first round hole feature comprises at least one of a first outer diameter, a first inner diameter, a first wall thickness; then as shown in fig. 6, step S301 includes:

S601, preprocessing a matching area to obtain the outer contour of each first round hole and the inner contour of each first round hole;

s602, determining a first outer diameter based on the outer contour of the first round hole, and/or determining a first inner diameter based on the inner contour of the first round hole, and/or determining a first wall thickness based on the outer contour of the first round hole and the inner contour of the first round hole.

Likewise, in some embodiments of the invention, the second round hole feature comprises at least one of a second outer diameter, a second inner diameter, a second wall thickness;

then, as shown in fig. 7, step S501 includes:

s701, preprocessing the second overlapping area to obtain the outer contour of each second round hole and the inner contour of each second round hole;

s702, determining the second outer diameter based on the outer contour of the second round hole, and/or determining the second inner diameter based on the inner contour of the second round hole, and/or determining the second wall thickness based on the outer contour of the second round hole and the inner contour of the second round hole.

In some embodiments of the present invention, the preprocessing in steps S601 and S701 includes, but is not limited to, image segmentation, image enhancement, and image contour extraction processes. Specifically: image segmentation is carried out on the first overlapped area based on an image segmentation algorithm, and the first overlapped area is segmented into a plurality of single round hole images, as shown in a left side diagram in fig. 8; then, sharpening (sharp) and histogram equalization (equalizhist) are carried out on each single round hole image, and round holes and backgrounds are displayed in a distinguishing mode, so that a clear round hole image can be obtained, and the clear round hole image is shown in the middle diagram in fig. 8; the outer and inner contours of each circular hole are then obtained by a contour extraction algorithm, as shown in the right-hand diagram of fig. 8.

As can be seen from fig. 8: the definition of the circular hole image outline can be improved through the pretreatment, so that the accuracy of the determined inner diameter, outer diameter and wall thickness can be improved.

Among them, the image segmentation algorithm includes, but is not limited to, find connected domain algorithm (findContours), template matching algorithm, and deep learning segmentation algorithm.

In some embodiments of the present invention, contour extraction algorithms include, but are not limited to: the outer contour and the inner contour are determined based on any contour extraction algorithm such as Canny algorithm, threshold segmentation algorithm, high-frequency information algorithm for extracting Fourier transform, ant colony algorithm and the like.

In some embodiments of the present invention, step S702 is specifically: extracting point coordinates of the outer contour, performing fitting calculation, and determining the outer diameter; extracting point coordinates of the inner contour, performing fitting calculation, and determining the inner diameter; the difference between the outer diameter and the inner diameter is the wall thickness. The extraction mode of the coordinates of the first round hole and the coordinates of the outer contour and the inner contour of the second round hole can be determined through a row and column number determining function (FindNonzero).

In some application scenarios, because the round hole may be blocked or not opened during processing, the round hole may appear in the first partial image or the second partial image after the camera shoots, the average gray values of the blocked round hole in the image are different, as shown in fig. 9, the round hole with the average gray value of 6 is the blocked round hole, and the round hole with the average gray value of 88 is the normal round hole. Thus, in some embodiments of the invention, the average gray value may be taken as a physical feature of a circular hole, namely: the first circular hole feature further comprises a first average gray value, and the second circular hole feature further comprises a second average gray value; step S301 further includes:

And determining a first round hole area of the first round hole based on the outer contour of the first round hole, and determining a first average gray value of the first round hole area.

Step S302 further includes:

and determining a second round hole area of the second round hole based on the outer contour of the second round hole, and determining a second average gray value of the second round hole area.

It should be understood that: when the round hole is blocked or the round hole is not opened, light cannot penetrate through the round hole, so that the average gray value of the round hole is low; and when the round hole is a through hole, the average gray value of the round hole is higher.

It should be noted that: the specific mode for determining the average gray value of the round hole area is as follows: firstly, acquiring gray values of all pixel points in a circular hole area, and secondly taking an average value of the gray values of all pixel points as an average gray value. For example: the circular hole area includes 4 pixel points, and the gray values of the pixel points are respectively 80, 82, 84 and 86, so that the average gray value of the circular hole is 83.

In the round hole manufacturing process, dirt can appear around some round holes in the structural member, and as shown in fig. 10, dirt does not exist around the round holes in the left side diagram, and the irregular shape around the round holes in the right side diagram is dirt. Thus, in some embodiments of the invention, the dirt values around the circular hole may also be taken as physical characteristics of the circular hole, namely: the first round hole feature further includes a first smudge value; then, as shown in fig. 11, step S301 further includes:

S1101, determining gray values of preset areas around the first round holes;

s1102, judging whether dirt exists in the peripheral preset area of the first round hole or not based on the gray value of the peripheral preset area of the first round hole;

s1103, when the gray value of the preset area around the first round hole is smaller than the gray threshold, dirt exists in the preset area around the first round hole, and the dirt value of the first round hole is set to be a first value;

s1104, when the gray value of the preset area around the first round hole is larger than or equal to the gray threshold value, no dirt exists in the preset area around the first round hole, and the dirt value of the first round hole is set to be a second value.

Likewise, the second hole feature further includes a second dirt value, and then as shown in fig. 12, step S501 further includes:

s1201, determining gray values of preset areas around the second round holes;

s1202, judging whether dirt exists in the peripheral preset area of the second round hole or not based on the gray value of the peripheral preset area of the second round hole;

s1203, when the gray value of the preset area around the second round hole is smaller than the gray threshold value, dirt exists in the preset area around the second round hole, and the dirt value of the second round hole is set to be a first value;

and S1204, when the gray value of the preset area around the second round hole is larger than or equal to the gray threshold value, no dirt exists in the preset area around the second round hole, and the dirt value of the second round hole is set to be a second value.

The gray threshold is set according to the background color, for example, the background color is bright, when the background color of the preset area around the hole is 255, the gray threshold can be set to be 255, and when the background color is dark, the threshold can be correspondingly set to be smaller.

It should be understood that: the peripheral preset range can be set or adjusted according to the size of the round hole and the distance between adjacent round holes, and similarly, the gray threshold can also be set or adjusted according to the actual application scene, and the method is not particularly limited.

In a specific embodiment of the present invention, the first value is 1 and the second value is 0.

In a specific embodiment of the present invention, if there is dirt at the circular holes of the second row and the third column in the first overlapping region and there is no dirt at the circular holes at other coordinates, the first circular hole dirt matrix Dirty _A The method comprises the following steps:

because the image stitching is performed through the round hole features with smaller orders of magnitude, the stitching efficiency is higher, and therefore when the image to be stitched comprises other partial images except the first partial image and the second partial image, the image stitching can be performed while the image is acquired, and the efficiency of generating the stitched image is improved. For convenience of explanation, the stitching process will be described by taking an example in which the image to be stitched includes a first partial image, a second partial image, and a third partial image.

In some embodiments of the present invention, as shown in fig. 13, the image stitching method includes:

s1301, acquiring a first partial image in real time based on a shooting camera, acquiring the characteristics of all round holes in a second partial image, determining a second round hole characteristic matrix based on the characteristics of all round holes in the second partial image and a second overlapping region, and determining a second round hole characteristic matrix based on the characteristics of all round holes in the second partial image and the second overlapping region;

s1302, splicing the first partial image and the second partial image based on the first circular hole feature matrix and the second circular hole feature matrix to obtain a partial spliced image, and simultaneously obtaining a third partial image based on a shooting camera in real time; wherein the second partial image includes a third overlapping region overlapping with the third partial image, the third partial image including a fourth overlapping region overlapping with the second partial image;

s1303, determining a reference area in a third overlapping area, determining a third round hole feature matrix of a round hole array in the reference area based on the round hole features of all the round holes in the second partial image and the reference area, and determining a fourth round hole feature matrix of the round hole array in a fourth overlapping area based on the round hole features of all the round holes in the third partial image and the fourth overlapping area;

And S1304, splicing the local spliced image with the third local image based on the third round hole feature matrix and the fourth round hole feature matrix to obtain a spliced image.

It should be noted that: the first partial image and the second partial image are images acquired at different positions on the structural member, so that the process of determining the first circular hole feature matrix can be performed in the moving process of moving the structural member from the position of the first partial image to the position of the second partial image, and the purposes of shooting and determining the circular hole feature matrix in one pass are realized. The method does not need to shoot the next local image after determining the round hole feature matrix of the local image, and improves the instantaneity of acquiring the spliced image.

The embodiment of the invention can acquire the third partial image in real time based on the shooting camera while splicing the first partial image and the second partial image, and then splice the third partial image and the partial spliced image to acquire the spliced image, thereby realizing that the spliced image can be acquired within a short period of time after the acquisition of the partial image, namely: the aim of image stitching is fulfilled while photographing (acquiring the partial images), and the acquired partial images are stitched after all the partial images are not required to be photographed, so that the instantaneity of acquiring the stitched images is further improved.

In a specific embodiment of the present invention, the specific process of acquiring and stitching the first partial image, the second partial image and the third partial image is as follows: firstly, acquiring a first local image; then, a second partial image is acquired, and a first round hole feature matrix is determined in the process of acquiring the second partial image; then determining a second round hole feature matrix, splicing the first local image and the second local image based on the first round hole feature matrix and the second round hole feature matrix, acquiring a third local image while splicing, determining a third round hole feature matrix and a fourth round hole feature matrix, and determining the third round hole feature matrix and the fourth round hole feature matrix while acquiring a local spliced image; and finally, splicing the local spliced image with the third local image based on the third round hole feature matrix and the fourth round hole feature matrix to obtain a spliced image.

The embodiment of the invention realizes the purposes of shooting and determining the circular hole feature matrix and splicing images at the same time, shortens the time from shooting local images and determining the circular hole feature matrix to splicing the images, and improves the efficiency of obtaining spliced images.

In some embodiments of the present invention, the second circular hole feature matrix comprises a plurality of second circular hole feature sub-matrices, wherein a matrix size of each second circular hole feature sub-matrix is the same as a matrix size of the first circular hole feature matrix. Then, as shown in fig. 14, step S103 includes:

s1401, determining inner diameter similarity, outer diameter similarity, wall thickness similarity, gray scale similarity and dirt similarity of a first circular hole feature matrix and each second circular hole feature sub-matrix;

s1402, acquiring a first weight of inner diameter similarity, a second weight of outer diameter similarity, a third weight of wall thickness similarity, a fourth weight of gray scale similarity and a fifth weight of dirt similarity;

s1403, determining the comprehensive similarity of the first circular hole feature matrix and each second circular hole feature submatrix based on the inner diameter similarity, the first weight, the outer diameter similarity, the second weight, the wall thickness similarity, the third weight, the gray scale similarity, the fourth weight, the dirt similarity and the fifth weight;

and S1404, taking the second round hole feature submatrix with the highest comprehensive similarity as a feature similarity matrix.

According to the embodiment of the invention, the integrated similarity determined based on the inner diameter similarity, the first weight, the outer diameter similarity, the second weight, the wall thickness similarity, the third weight, the gray level similarity, the fourth weight, the dirt similarity and the fifth weight is used as the basis for determining the feature similarity matrix, so that the reliability of the selected feature similarity matrix can be improved, and the accuracy and the reliability of image splicing are further improved.

It should be understood that: the sum of the first weight, the second weight, the third weight, the fourth weight and the fifth weight is 1, and specific numerical values of the first weight, the second weight, the third weight, the fourth weight and the fifth weight should be set according to experience values. In a specific embodiment of the present invention, the first weight, the second weight, the third weight, the fourth weight, and the fifth weight are all 0.2.

It should be noted that: the similarity of the physical features of the first circular hole feature matrix and the feature similarity matrix in each circular hole should be approximately the same, and when one of the inner diameter similarity, the outer diameter similarity, the wall thickness similarity, the gray level similarity or the dirt similarity of the feature similarity matrix and the first circular hole feature matrix is larger than the remaining four similarities, the calculation of the comprehensive similarity should be stopped, the abnormality is recorded, and the unified processing of the user is waited. Therefore, errors in image splicing caused by incorrect circular hole feature matrix are avoided, and the accuracy and reliability of the image splicing method are further improved.

Also to be described is: the inner diameter similarity, the outer diameter similarity, the wall thickness similarity, the gray level similarity and the dirt similarity can be obtained through similarity evaluation indexes, wherein the similarity evaluation indexes comprise, but are not limited to, sum of square differences, normalized square differences, correlation coefficients and the like.

Specifically, the sum of squares is calculated as:

the normalized square error calculation formula is:

the correlation calculation formula is:

the correlation coefficient calculation formula is:

in the method, in the process of the invention,is one of inner diameter similarity, outer diameter similarity, wall thickness similarity, gray level similarity and dirt similarity; />One of the inner diameter, the outer diameter, the wall thickness, the average gray value and the dirt value in the second round hole feature sub-matrix; />One of the inner diameter, the outer diameter, the wall thickness, the average gray value and the dirt value in the first round hole feature matrix; />The coordinate vector of each round hole in the first round hole feature matrix; />Is the offset vector of the second circular hole feature sub-matrix.

In some embodiments of the present invention, there are a plurality of overlapping points in the first circular hole feature matrix and the feature similarity matrix, and coordinates of the overlapping points in the first overlapping region and coordinates of the overlapping points in the second overlapping region are obtained, respectively, that is: obtaining N sets of point pairs, step S104 specifically includes:

determining an affine transformation matrix M based on the N groups of point pairs, wherein the affine transformation matrix M meets the following conditions:

P _B ×M＝P _A

in the method, in the process of the invention,P _A coordinates of the overlap point in the first overlap region; p (P) _B Is the coordinates of the overlapping point in the second overlapping region.

After the affine transformation matrix M is obtained, the second partial image can be subjected to radial change, and the second partial image is transformed into the image coordinate system of the first partial image, so that the first partial image and the second partial image can be spliced in the image coordinate system of the first partial image.

It should be noted that: when the image to be spliced comprises a plurality of partial images, all the partial images are spliced to generate a spliced image with a huge size, in some specific scenes, the size of the spliced image is larger than 10G, for convenient storage, reading and display, in some embodiments of the invention, all the partial images after affine transformation are stored separately according to the sequence, when a worker needs to splice part or all the images, all the partial images stored separately are directly called and spliced, and the spliced image is displayed.

In a specific application scenario, quality detection is required to be performed on the spliced image after splicing, wherein the quality detection includes an inner diameter, an outer diameter, a wall thickness, an average gray value and a dirty value of each round hole, and physical characteristic values of the inner diameter, the outer diameter, the wall thickness, the average gray value and the dirty value of each round hole are also used in the image splicing process, so in order to improve quality detection efficiency, in some embodiments of the present invention, a first round hole characteristic matrix and a second round hole characteristic matrix may be determined in step S102, and a third round hole characteristic matrix of other areas except for the first overlapping area in the first partial image and a fourth round hole characteristic matrix of other areas except for the second overlapping area in the second partial image may be determined, so that quality of the spliced image may be determined directly according to the first round hole characteristic matrix, the third round hole characteristic matrix and the fourth round hole characteristic matrix of the first overlapping area after determining the spliced image, and quality detection efficiency of the structural member may be improved.

In order to better implement the image stitching method in the embodiment of the present invention, correspondingly, as shown in fig. 15, the embodiment of the present invention further provides a spacer image reconstruction device, where the image stitching device 1500 includes:

the image to be stitched acquiring unit 1501 is configured to acquire an image to be stitched, where the image to be stitched includes a first partial image and a second partial image, the first partial image includes a first overlapping region overlapping with the second partial image, and the second partial image includes a second overlapping region overlapping with the first partial image;

a feature matrix determining unit 1502 configured to determine a matching region in the first overlapping region, and determine a first circular hole feature matrix of the circular hole array in the matching region and a second circular hole feature matrix of the circular hole array in the second overlapping region;

the feature similarity matrix determining unit 1503 is configured to traverse the second circular hole feature matrix based on the first circular hole feature matrix, and determine a feature similarity matrix with the highest similarity with the first circular hole feature matrix in the second circular hole feature matrix;

an image stitching unit 1504 is configured to stitch the first partial image and the second partial image based on the first circular hole feature matrix and the feature similarity matrix.

The image stitching device 1500 provided in the foregoing embodiment may implement the technical solutions described in the foregoing image stitching method embodiments, and the specific implementation principles of the foregoing modules or units may be referred to the corresponding content in the foregoing image stitching method embodiments, which is not described herein again.

The invention also provides a glass through hole image positioning method, which comprises the following steps:

shooting a glass through hole product, acquiring a plurality of glass through hole (TGV) images, and splicing the plurality of glass through hole images based on an image splicing method; positioning bad points in the spliced image;

the image stitching method is a method described in the above image stitching method embodiment, and is referred to the corresponding content in the above image stitching method embodiment, which is not described herein again.

Further, the pore diameter of each round hole in the glass through hole product is 20-100 μm, the wall thickness of the round hole is 0.5-5 μm, and the glass is led throughThe hole product comprises a plurality of round holes with similar sizes and shapes, the round holes are regularly arranged in rows and columns, and the hole density is at least 500-1000 holes/mm ² The aperture of the round hole is extremely small, so that the difficulty of the image splicing method is increased, the surface of a glass through hole product is smooth, the color of glass is single, and the conventional image splicing method cannot accurately and efficiently finish splicing. According to the embodiment of the invention, the image splicing method is used for splicing the glass through hole images, so that accurate and efficient splicing can be realized, and further, the positioning accuracy rate of the defective points in the spliced images is improved when the defective points are positioned, and an operator or a mechanical arm is guided to process the positioned defective points.

As shown in fig. 16, the present invention further provides an image stitching device 1600 accordingly. The image stitching device 1600 includes a processor 1601, a memory 1602, and a display 1603. Fig. 16 shows only some of the components of the image stitching device 1600, but it should be understood that not all of the illustrated components are required to be implemented and that more or fewer components may alternatively be implemented.

The memory 1602 may in some embodiments be an internal storage unit of the image stitching device 1600, such as a hard disk or memory of the image stitching device 1600. The memory 1602 may also be an external storage device of the image stitching device 1600 in other embodiments, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc. provided on the image stitching device 1600. Further, the memory 1602 may also include both internal and external storage units of the image stitching device 1600. The memory 1602 is used to store application software and various types of data for installing the image stitching device 1600.

The processor 1601 may be a central processing unit, microprocessor or other data processing chip in some embodiments for executing program code or processing data stored in the memory 1602, such as the image stitching method of the present invention.

The display 1603 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an organic light emitting diode touch, or the like in some embodiments. The display 1603 is for displaying information at the image stitching device 1600 and for displaying a visual user interface. The components 1601-1603 of the image stitching device 1600 communicate with each other over a system bus.

In some embodiments of the present invention, when the processor 1601 executes the image stitching program in the memory 1602, the following steps may be implemented:

acquiring an image to be stitched, wherein the image to be stitched comprises a first local image and a second local image, the first local image comprises a first overlapping area overlapped with the second local image, and the second local image comprises a second overlapping area overlapped with the first local image;

determining a matching area in the first overlapping area, and determining a first round hole feature matrix of a round hole array in the matching area and a second round hole feature matrix of a round hole array in the second overlapping area;

traversing a second round hole feature matrix based on the first round hole feature matrix, and determining a feature similarity matrix with highest similarity with the first round hole feature matrix in the second round hole feature matrix;

It should be understood that: the processor 1601 may perform other functions in addition to the above functions when executing the tile image reconstruction program in the memory 1602, see in particular the description of the corresponding method embodiments above.

Correspondingly, the embodiment of the application also provides a computer readable storage medium, and the computer readable storage medium is used for storing a computer readable program or instruction, and when the program or instruction is executed by a processor, the steps or functions of the image stitching method provided by the above method embodiments can be realized.

Those skilled in the art will appreciate that all or part of the flow of the methods of the embodiments described above may be accomplished by way of a computer program stored in a computer readable storage medium to instruct related hardware (e.g., a processor, a controller, etc.). The computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory.

The image stitching method, the device, the glass through hole image positioning method and the glass through hole image positioning equipment provided by the application are described in detail, and specific examples are applied to illustrate the principle and the implementation mode of the application, and the description of the above examples is only used for helping to understand the method and the core idea of the application; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in light of the ideas of the present application, the present description should not be construed as limiting the present application.

Claims

1. An image stitching method, which is used for stitching images to be stitched with a circular hole array, comprises the following steps:

2. The image stitching method of claim 1 wherein the first round hole feature matrix comprises at least one of a first round hole outside diameter matrix, a first round hole inside diameter matrix, a first round hole wall thickness matrix, a first round hole gray scale matrix, and a first round hole dirt matrix; the second round hole feature matrix comprises at least one of a second round hole outer diameter matrix, a second round hole inner diameter matrix, a second round hole wall thickness matrix, a second round hole gray scale matrix and a second round hole dirt matrix.

3. The method of image stitching according to claim 1, wherein the determining a first circular aperture feature matrix of an array of circular apertures in the matching region comprises:

4. The image stitching method of claim 3 wherein the first circular hole feature comprises at least one of a first outer diameter, a first inner diameter, and a first wall thickness and the second circular hole feature comprises at least one of a second outer diameter, a second inner diameter, and a second wall thickness;

5. The image stitching method of claim 4 wherein the first circular aperture feature further comprises a first average gray value and/or a first smudge value and the second circular aperture feature further comprises a second average gray value and/or a second smudge value;

and/or the number of the groups of groups,

determining gray values of preset areas around the first round holes;

And/or the number of the groups of groups,

determining gray values of preset areas around the second round holes;

6. The image stitching method according to claim 1, wherein the image to be stitched further comprises a third partial image, the image stitching method comprising:

7. The image stitching method of claim 1 wherein the second circular aperture feature matrix comprises a plurality of second circular aperture feature sub-matrices; the determining the feature similarity matrix with the highest similarity with the first round hole feature matrix in the second round hole feature matrix comprises the following steps:

8. An image stitching device for stitching images to be stitched having an array of circular holes, comprising:

9. A glass through-hole image positioning method, characterized by comprising:

shooting a glass through hole product, obtaining a plurality of glass through hole images, and splicing the plurality of glass through hole images based on an image splicing method to obtain spliced images;

positioning bad points in the spliced image;

wherein the image stitching method is the image stitching method according to any one of claims 1 to 7.

10. The image stitching method according to claim 9, wherein the aperture of each circular hole in the glass through-hole product is 20 μm to 100 μm.

11. An image stitching device comprising a memory and a processor, wherein,

the memory is used for storing programs;

the processor, coupled to the memory, is configured to execute the program stored in the memory to implement the steps in the image stitching method of any of the above claims 1 to 7.