CN116091404A

CN116091404A - Welding line defect detection and pattern recognition method based on image-point cloud information fusion

Info

Publication number: CN116091404A
Application number: CN202211522237.7A
Authority: CN
Inventors: 于乃功; 李洪政; 徐乔
Original assignee: Beijing University of Technology
Current assignee: Beijing University of Technology
Priority date: 2022-12-01
Filing date: 2022-12-01
Publication date: 2023-05-09

Abstract

The invention provides a bonding wire defect detection and pattern recognition method based on image-point cloud information fusion, which can detect a defective bonding wire in a PCB and judge the defect pattern of the defective bonding wire. The method belongs to the defect detection field in the wire bonding manufacturing process, and aims to solve the problems of high labor intensity, low detection efficiency and the like in the existing defect detection method. The specific detection flow comprises the following steps: acquiring a depth image of a PCB, judging a welding line contour in the depth image through an OBB bounding box and an MAD algorithm, extracting depth data in the contour, and converting the depth data into point cloud to obtain point cloud data of the welding line; performing point cloud pretreatment and surface segmentation to finish accurate extraction of the welding line surface; constructing a defect classification model to finish detection of bonding wire defects and defect mode identification; and finally, calculating the curvature of the normal defect welding line to finish curvature evaluation of the normal welding line.

Description

Welding line defect detection and pattern recognition method based on image-point cloud information fusion

Technical field:

the invention belongs to the field of defect detection in the wire bonding production and manufacturing process. In particular to a method for detecting and judging the type of a defect mode by adopting an image-point cloud information fusion method for the defects of a bonding wire structure in lead welding.

Background

In the key production link of the semiconductor, the problem of detecting the quality of wire bonding still has some problems to be solved in a urgent way. The wire bonding technique is a process of connecting a first solder joint and a corresponding second solder joint using a metal wire. The defect of the bonding wire structure is a main factor influencing the function of the product and the yield of the product. By detecting whether the bonding wires have structural defects, the quality problem of the product bonding wires can be found, defect mode identification can be carried out according to the defective bonding wires, faults existing in the process flow can be qualitatively assessed, and engineers can conveniently carry out process improvement. Currently, wire bond defect detection is mainly divided into two categories: and detecting the structural defect of the welding wire by manual visual inspection and detecting the broken wire defect of the welding wire by image processing. The manual visual inspection mode has low efficiency, low speed and high labor intensity, and the defect mode which can be detected by image processing is single, so that a method capable of automatically detecting the defects of the bonding wire structure and identifying the defect mode is needed.

For the defect detection of bonding wires, at present, methods based on image processing, such as threshold segmentation, region growth, connected domain detection and the like, are adopted. The method relies on a traditional camera to collect a plane image, and can only detect the broken line defects of the welding line, and can not finish the detection of other types of defects of the welding line and the identification of defect modes, such as bending, line collapse, abnormal radian and the like.

Disclosure of Invention

The invention mainly aims to provide a bonding wire defect detection and pattern recognition method based on image-point cloud information fusion, which can be applied to the manufacturing process of wire bonding, replaces the existing manual visual inspection mode, realizes automatic detection of the quality problem of wire bonding in products by a machine, performs pattern recognition according to defective bonding wires, and completes classification of defective bonding wires. The present invention aims to solve the following problems:

1. the defect detection of the welding wire mainly adopts manual visual inspection, and has low efficiency, low speed and high labor intensity;

2. the existing bonding wire defect detection technology can only detect broken wires and cannot detect multiple types of defects such as bending and collapsing of the bonding wires;

3. the existing bonding wire defect detection method cannot complete the identification of multiple types of defect modes, and cannot give quantitative evaluation indexes aiming at normal bonding wires, so that the calculation of the radian of the bonding wires cannot be completed.

In order to solve the problems, the invention provides a bonding wire defect detection and pattern recognition method based on image-point cloud information fusion. And detecting defects such as broken lines, collapsed lines and bending on the divided welding line planes respectively, and calculating the space curvature and the curvature of the fitting mathematical model on the normal welding line.

The specific working flow of the invention is as follows:

step 1, scanning the whole PCB by using a 3D laser contour sensor to obtain a depth image;

step 2, dividing the PCB image and the background image by adopting methods of image enhancement, threshold segmentation and contour detection to obtain a depth image of the PCB;

step 3, carrying out image enhancement and threshold segmentation on the depth map of the PCB, carrying out contour detection on the image subjected to threshold segmentation, traversing and calculating an OBB bounding box of each contour, obtaining the length-width ratio of the bounding box, and distinguishing the welding line contour and other contours by adopting an absolute medium bit difference (MAD) algorithm;

step 4, extracting depth data in the outline of the welding line, performing matrix operation, obtaining point cloud data of the welding line, and preprocessing the point cloud data by adopting statistical filtering and PCA affine transformation;

step 5, carrying out curved surface segmentation on the preprocessed welding line data to complete segmentation of welding line curved surface data and redundant data and obtain complete welding line point cloud data;

step 6, projecting the welding line point cloud data constructed in the step 5 to an X-Z axis, calculating the size of the interval between adjacent points after projection, and adopting a 3-sigma algorithm to finish screening of in-line broken lines so as to obtain a non-broken line welding line point cloud data set;

step 7, performing FPS downsampling on the welding line point cloud data constructed in the step 6, completing estimation of normal vectors of each point in the downsampled data, constructing a defect classification model, and completing identification of a specific welding line defect mode;

and 8, calculating the space curvature of the normal welding line in the step 7, projecting the welding line data to the X-Z axis, fitting the mathematical model of the welding line by adopting a least square method, calculating the curvature of the fitting mathematical model, and finishing the evaluation of the curvature of the normal welding line.

In step 3, the OBB bounding box aspect ratio calculation and the wire bonding contour screening process are as follows, the contrast enhancement and sharpening process is performed on the PCB depth image, and the enhanced image threshold is segmented. And carrying out contour detection on the binarized image subjected to threshold segmentation, analyzing and calculating the characteristic value and the characteristic vector of each group of contour data by adopting PCA principal component, determining the characteristic vector corresponding to the calculated maximum characteristic value as an X axis, and carrying out affine transformation on the contour data to a new principal axis. Calculating maximum and minimum values of X axis and Y axis in affine transformation data, fitting an outline circumscribed rectangular frame, calculating length-width ratio of the rectangular frame, and marking as D _i . The aspect ratio of all contours is calculated by traversing, and an aspect ratio data set D is generated.

Calculating the median of the data set D, denoted as D _median Subtracting the median from the data set D to obtain a new set D ', calculating the median of D ', and marking the median as D ' _median . D satisfying the formula (1) _i Is the aspect ratio of the profile of the wire bond.

D _i -D _median >8*D' _median (1)

In step 5, slicing the point cloud data preprocessed in step 4 along the X axis, wherein the slice thickness is the minimum interval between adjacent points of the X axis. Extracting all space points belonging to the same X scale, wherein the space points are distributed in the same Y-Z axis plane, drawing histograms of the extracted points at different heights of the Z axis, and marking the points with the largest distribution quantity at the same height as central points.

Traversing to calculate the distance l between the center point Pn and the rest of the points Pi _i Angle alpha of Y-axis _i And the normal vector included angle theta _i Generates a descriptor set (l _i ,α _i ,θ _i ). If three descriptors in Pi satisfy any two conditions in equation (2), equation (3) and equation (4), then the point is added to the set of center points. And traversing and calculating a center point set under each X-axis scale, and combining all the center point sets to obtain a final segmented weld line Qu Miandian cloud.

α<5° (3)

θ<5° (4)

Middle l _μ 、l _σ The mean and variance of the distance descriptors, respectively.

In step 6, performing FPS downsampling on the wire bonding point cloud, where each group of downsampled wire bonding data has M points. And establishing a KD-Tree Tree structure for the down-sampling point cloud, randomly selecting a seed point from the first 5 points along the X-axis direction, selecting a seed point every K points, and selecting N seed points in total, wherein N is a positive integer. And inquiring K adjacent points near the seed point by utilizing the KD-Tree to form N local planes, calculating the normal vector of each point in the local planes, and marking the normal vector average value of the fitting planes as the normal vector of the planes. And calculating the included angle between the normal vector of the plane and the X axis. The included angle is larger than 90 degrees, the local plane descriptor is recorded as 1, and conversely, the local plane descriptor is recorded as-1. And completing the calculation of N local plane descriptors to generate a descriptor set Q.

And (3) expressing part of the description subsets in the description subset generated in the step (1) as a formula (6). After the generation of the descriptor set, the defect mode is identified by looking up table 1.

The invention has the following advantages:

the method adopts the image-point cloud information fusion mode to detect the structural defects of the welding wires, thereby greatly reducing the labor cost and improving the welding wire detection efficiency. The method can also perform pattern recognition on the defect of the bonding wire, and is convenient for subsequent process evaluation and improvement. And (3) finishing coarse positioning of the welding line by adopting an OBB bounding box and an MAD algorithm, and performing matrix operation on the positioning information to obtain the point cloud information of the welding line. Aiming at obtaining point cloud information, a surface feature descriptor of a point with the maximum space density under a point cloud slice is provided, the point cloud can be subjected to space plane segmentation, redundant information is deleted, an accurate welding line plane is obtained, and detection of various defects of a welding line is more conveniently completed. The invention provides a normal vector direction differential descriptor based on multi-local plane fitting on the basis of wire bonding point cloud data segmentation, which is used for wire bonding defect detection and pattern recognition.

Drawings

FIG. 1 is a flow chart of wire bond defect detection;

fig. 2 (a) is a depth image acquired by a camera, (b) is a depth image of a PCB board, (c) is an image after threshold segmentation, and (d) is an image after contour detection and drawing of an OBB bounding box;

fig. 3 (a) is a point cloud before affine transformation, and (b) is a point cloud after affine transformation;

fig. 4 (a) shows a point cloud before surface segmentation and (b) shows a point cloud after surface segmentation;

fig. 5 (a) shows a normal wire, (b) shows a bending defect, (c) shows a collapse defect, (d) shows a single-ended wire break, (e) shows an in-line wire break, and (f) shows an arc anomaly.

Detailed Description

The following describes the method in detail with reference to the drawings and examples.

Fig. 1 is a flowchart of a method for detecting and identifying a bonding wire defect, firstly, a depth image of a PCB board is obtained, and positioning of a bonding wire contour is completed in the depth image. And converting the welding line depth map into point cloud data, and carrying out plane segmentation on the welding line point cloud to obtain a complete welding line plane. And detecting defects such as broken lines, collapsed lines and bending on the divided welding line planes respectively, and calculating the space curvature and the curvature of the fitting mathematical model on the normal welding line.

The method comprises the following specific steps:

1. collecting depth map of PCB

The image acquisition equipment adopts a 3D laser contour sensor of the sea-health robot, the resolution of an X axis is 30 micrometers, the field of view of a far vision is 90 millimeters, and the acquired depth image can be transmitted to a computer for processing in real time.

2. Welding wire contour positioning

The acquisition of the depth map of the PCB is shown in fig. 2 (a), the segmentation of the image of the PCB and the image background is carried out firstly to obtain the complete depth map of the PCB, the image enhancement and the threshold segmentation are carried out on the segmented image of the PCB, and the segmentation effect is shown in fig. 2 (c).

And (3) carrying out contour detection on the figure 2 (c), extracting each contour data, adopting PCA principal component analysis to calculate the characteristic value and the characteristic vector of each group of contour data, determining the characteristic vector corresponding to the calculated maximum characteristic value as an X axis, and carrying out affine transformation on the contour data to a new principal axis. Calculating maximum and minimum values of X axis and Y axis in affine transformed data, fitting an outline circumscribed rectangular frame, calculating aspect ratio of rectangular frame as shown in FIG. 2 (D), and recording as D _i . The aspect ratio of all contours is calculated by traversing, and an aspect ratio data set D is generated.

Calculating the median of the data set D, denoted as D _median Subtracting the median from the data set D to obtain a new set D ', calculating the median of D ', and marking the median as D ' _median . D satisfying the formula (7) _i Is the aspect ratio of the profile of the wire bond.

D _i -D _median >8*D' _median (7)

And (3) extracting data in the outline of the welding line, and converting the depth data into point cloud through a formula (8).

Wherein X, Y, Z is the coordinate of the point cloud in space, X and y are the coordinates of the pixel points in the depth map, z is the height information in the depth map, and X _scale 、Y _scale 、Z _scale 、X _offset 、Y _offset 、Z _offset Is a parameter inside the camera.

3. Curved surface segmentation

And firstly carrying out statistical filtering on the extracted welding line point cloud, wherein the number of neighborhood points used for designating calculation average distance in filtering parameters is 200, and the standard deviation of the average distance of the point cloud is 2.0. And (3) carrying out principal component analysis on the filtered point clouds, calculating characteristic values and characteristic vectors of each group of point clouds, determining the characteristic vector corresponding to the calculated maximum characteristic value as an X axis, and carrying out affine transformation on the point cloud space coordinates (X, Y, Z) by adopting a formula (9), wherein θ is an included angle between a new X axis and an original X axis, so as to obtain point clouds (X ', Y ', Z ') after affine transformation. The point cloud space pose before and after affine transformation is shown in fig. 3.

And slicing the point cloud data after affine transformation along the X axis, wherein the slice thickness is the minimum interval between adjacent points of the X axis. Extracting all space points belonging to the same X scale, wherein the space points are distributed in the same Y-Z axis plane, drawing histograms of the extracted points at different heights of the Z axis, and marking the points with the largest distribution quantity at the same height as central points. Traversing to calculate the distance l between the center point Pn and the rest of the points Pi _i Angle alpha of Y-axis _i And the normal vector included angle theta _i Generates a descriptor set (l _i ,α _i ,θ _i ). If three descriptors in Pi are arbitrarily fullGiven the two conditions in equation (10), equation (11) and equation (12), this point is added to the set of center points.

α _i <5° (11)

θ _i <5° (12)

And traversing and calculating a center point set under each X-axis scale, and combining all the center point sets to obtain a final segmented weld line Qu Miandian cloud. The front and back effects of point cloud surface segmentation are shown in fig. 4.

4. Wire bond defect detection

Projecting the segmented Qu Miandian cloud to an X-Z axis, and calculating the X-axis distance between adjacent points by using a formula (13):

E _i ＝x _i -x _i-1 (13)

calculating the average value of the spacing:

calculating the variance of the spacing:

if the distance E between adjacent points _i Satisfying equation (14) indicates that the bonding wire is an in-line break.

E _i >u+3σ (16)

In the formula (13), x _i 、x _i-1 X-axis coordinates of the point cloud, E _i Is the distance between the X axes of adjacent point clouds; μ in formula (14) is an average value of the pitches; in the formula (15), σ is the variance of the pitch.

If the bonding wire is not broken in-line, performing FPS downsampling on the bonding wire point cloud, wherein each group of downsampled bonding wire data has M points. And establishing a KD-Tree Tree structure for the down-sampling point cloud, randomly selecting a seed point from the first 5 points along the X-axis direction, selecting a seed point every K points, and selecting N seed points in total, wherein N is a positive integer. And inquiring K adjacent points near the seed point by utilizing the KD-Tree to form N local planes, calculating the normal vector of each point in the local planes, and marking the normal vector average value of the fitting planes as the normal vector of the planes. And calculating the included angle between the normal vector of the plane and the X axis. The included angle is larger than 90 degrees, the local plane descriptor is recorded as 1, and conversely, the local plane descriptor is recorded as-1. And completing the calculation of N local plane descriptors to generate a descriptor set Q.

Writing a part of the subset of descriptions in the subset of descriptions generated in (1) as formula (18). After the generation of the descriptor set, the defect mode is identified by looking up table 1.

Wherein N is the number of descriptors, i and j are the number of descriptors 1 or-1, and N and m are the number of i and j respectively.

Table 1 depicts a sub-defect pattern comparison table

6. Welding line curvature calculation

And after the defect detection of the bonding wire is finished, obtaining the normal bonding wire. For normal welding line point cloud data, firstly establishing a KD-Tree Tree structure, inquiring n adjacent points Pk of points Pi in the point cloud, establishing a adjacent point set X, decentering the set X, and constructing a covariance matrix. SVD (singular value decomposition) is carried out on the covariance matrix to obtain a characteristic value set lambda= { lambda ₁ ,λ ₂ ,λ ₃ Calculating curvature by using a formula (19), and traversing to calculate the curvature of all space points to obtain a curvature set.

{ lambda } in formula (19) ₁ ,λ ₂ ,λ ₃ The characteristic values after covariance matrix decomposition are sequentially arranged from small to large. Delta is the calculated spatial curvature.

And projecting the data to the X-Z axis for the normal welding line point cloud data, performing data fitting on the projected welding line by adopting a least square method to obtain a corresponding mathematical model, and calculating the curvature of the mathematical model. Fitting model f (x) =w ₀ +w ₁ x+w ₂ x ² +w ₃ x ³ The fitting condition is formula (20), and the curvature calculation method is formula (21).

/>

F (x) in formula (20) is a fitting model, z _i Z-axis data, w, of point cloud _i For the coefficients to be determined; in equation (21), K is the calculated curvature, and f', f″ are the first and second derivatives of the model f (x).

Claims

1. A bonding wire defect detection and pattern recognition method based on image-point cloud information fusion is characterized by comprising the following steps:

2. The method for detecting and identifying the defects of the bonding wires based on the fusion of image-point cloud information as set forth in claim 1, wherein the step 3 is specifically as follows:

(1) Contour circumscribed rectangular frame length-width ratio calculation

Performing image enhancement and threshold segmentation on the PCB depth image, and performing contour detection on the binarized image after threshold segmentation; adopting PCA principal component analysis to calculate the characteristic value and characteristic vector of each group of contour data, determining the characteristic vector corresponding to the calculated maximum characteristic value as an X axis, and affine transforming the contour data to a new principal axis; calculating maximum and minimum values of X axis and Y axis in affine transformation data, fitting an outline circumscribed rectangular frame, calculating length-width ratio of the rectangular frame, and marking as D _i The method comprises the steps of carrying out a first treatment on the surface of the Traversing and calculating the length-width ratio of all the contours to generate a length-width ratio data set D;

(2) Welding wire contour positioning

Calculating the median of the data set D, denoted as D _median Subtracting the median from the data set D to obtain a new set D ', calculating the median of D ', and marking the median as D ' _median The method comprises the steps of carrying out a first treatment on the surface of the D satisfying the formula (1) _i Aspect ratio of the profile of the wire;

D _i -D _median >8*D' _median (1)。

3. the method for detecting and identifying the defects of the bonding wires based on the fusion of the image-point cloud information as set forth in claim 1, wherein the step 5 is specifically as follows:

(1) Center point calculation

Slicing the point cloud data preprocessed in the step 4 along the X axis, wherein the slice thickness is the minimum interval between adjacent points of the X axis; extracting all space points belonging to the same X scale, wherein the space points are distributed in the same Y-Z axis plane, drawing histograms of the extracted points at different heights of the Z axis, and marking the points with the largest distribution quantity at the same height as central points;

(2) Point cloud descriptor computation

Traversing to calculate the distance l between the center point Pn and the rest of the points Pi _i Angle alpha of Y-axis _i And the normal vector included angle theta _i Generates a descriptor set (l _i ,α _i ,θ _i ) The method comprises the steps of carrying out a first treatment on the surface of the If three descriptors in Pi meet two conditions in the formula (2), the formula (3) and the formula (4) at random, adding the point into a center point set;

α _i <5° (3)

θ _i <5° (4)

middle l _μ 、l _σ Respectively mean and variance of the distance descriptors;

(3) Bonding wire curved surface extraction

And (3) traversing and calculating the center point set under each X-axis scale by adopting the methods in (1) and (2), and combining all the center point sets to obtain the final segmented weld line Qu Miandian cloud.

4. The method for detecting and identifying the defects of the bonding wires based on the fusion of the image-point cloud information as set forth in claim 1, wherein the step 7 is specifically as follows:

(1) Wire bonding defect descriptor calculation

Performing FPS downsampling on the welding line point cloud, wherein each group of downsampled welding line data has M points; establishing a KD-Tree Tree structure for the down-sampling point cloud, randomly selecting a seed point from the first 5 points along the X-axis direction, selecting a seed point every K points, and selecting N seed points in total, wherein N is a positive integer; inquiring K adjacent points near the seed point by utilizing the KD-Tree to form N local planes, calculating the normal vector of each point in the local planes, fitting the normal vector mean value of the planes, and recording as the plane normal vector; calculating an included angle between the normal vector of the plane and the X axis; the included angle is larger than 90 degrees, the local plane descriptor is recorded as 1, and conversely, the local plane descriptor is recorded as-1; completing the calculation of N local plane descriptors to generate a descriptor set Q;

(2) Defect pattern recognition

Writing a part of the description subset in the description subset generated in the step (1) as a formula (6); after the generation of the description subset, completing the identification of the defect mode by looking up the table 1;

wherein N is the number of descriptors, i and j are the number of descriptors 1 or-1, and N and m are the number of i and j respectively;

table 1 depicts a sub-defect pattern comparison table

/>