CN113538341B - Automatic optical detection assisting method, device and storage medium - Google Patents

Automatic optical detection assisting method, device and storage medium Download PDF

Info

Publication number
CN113538341B
CN113538341B CN202110707665.6A CN202110707665A CN113538341B CN 113538341 B CN113538341 B CN 113538341B CN 202110707665 A CN202110707665 A CN 202110707665A CN 113538341 B CN113538341 B CN 113538341B
Authority
CN
China
Prior art keywords
picture
rechecking
model
pcb
sample
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110707665.6A
Other languages
Chinese (zh)
Other versions
CN113538341A (en
Inventor
陆唯佳
龚昊
刘鹏
李兵洋
葛欢
张洁
王�琦
金昱
彭社长
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
United Automotive Electronic Systems Co Ltd
Original Assignee
United Automotive Electronic Systems Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by United Automotive Electronic Systems Co Ltd filed Critical United Automotive Electronic Systems Co Ltd
Publication of CN113538341A publication Critical patent/CN113538341A/en
Application granted granted Critical
Publication of CN113538341B publication Critical patent/CN113538341B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30141—Printed circuit board [PCB]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

The application discloses an automatic optical detection auxiliary method, an automatic optical detection auxiliary device and a storage medium, and relates to the technical field of automatic detection. The automatic optical detection auxiliary method comprises the steps of obtaining res files output by AOI equipment; analyzing the res file, and storing the PCB picture and the picture imaging algorithm contained in the res file as a file in a preset format; preprocessing the PCB picture to obtain a target picture with a preset size; selecting a rechecking model according to a picture imaging algorithm corresponding to the PCB picture, inputting the target picture into the rechecking model to obtain a rechecking result of the detection point, wherein the rechecking result comprises the target picture and a label; the rechecking model is obtained by training a deep learning algorithm, and the label is a defective part or a good product; the problems of high false positive rate, low manual rechecking efficiency and high cost of the detection result of the existing AOI equipment are solved; the effects of reducing the workload of visual inspection and improving the detection accuracy are achieved.

Description

Automatic optical detection assisting method, device and storage medium
Technical Field
The application relates to the technical field of automatic detection, in particular to an automatic optical detection auxiliary method, an automatic optical detection auxiliary device and a storage medium.
Background
With the continuous development of science, technology and economy, the demands of people for electronic products are also increasing, and the application of PCB (printed circuit board ) products is also increasing.
The processing of PCB products typically involves a reflow soldering process for the chip components and a selective wave soldering process for the discrete package components. After welding, in order to ensure the quality of the PCB product, the welded PCB product needs to be detected. Currently, AOI (Automated Optical Inspection ) replaces most of the manual inspection. Line-side AOI equipment is typically used to detect possible welding defects on a production line, such as: foot lifting, part missing, bridging, reverse pasting and the like. Conventional AOI equipment suppliers, such asConventional machine vision inspection techniques are used, and manual features are largely used as inspection basis.
However, the output result of the detection algorithm in the traditional AOI equipment has high false positive rate and serious false positive, and also requires a visual inspection worker to recheck suspected defect parts, so that a large amount of human resources are consumed, and the detection speed is low.
Disclosure of Invention
In order to solve the problems in the related art, the application provides an automatic optical detection assisting method, an automatic optical detection assisting device and a storage medium. The technical scheme is as follows:
in a first aspect, an embodiment of the present application provides an automatic optical detection assistance method, including:
acquiring res files output by the AOI equipment, wherein the res files correspond to the detection points;
analyzing the res file, and storing the PCB picture, the metadata and the picture imaging algorithm contained in the res file as files in a preset format;
Preprocessing the PCB picture to obtain a target picture with a preset size; the target picture corresponds to a part in the AOI detection frame in the PCB picture;
Selecting a rechecking model according to a picture imaging algorithm corresponding to the PCB picture, inputting the target picture into the rechecking model to obtain a rechecking result of the detection point, wherein the rechecking result comprises the target picture and a label; the rechecking model is obtained by training a deep learning algorithm, and the label is a defective part or a good product.
Analyzing the res file by acquiring the res file output by the AOI equipment, storing the PCB picture, the metadata and the picture imaging algorithm contained in the res file as files in a preset format, obtaining a target picture with a preset size before processing the PCB picture, selecting a rechecking model according to the picture imaging algorithm corresponding to the PCB picture, and inputting the target picture into the rechecking model to obtain a rechecking result of the detection point; the problems of high false positive rate, low manual rechecking efficiency and high cost of the detection result of the existing AOI equipment are solved; the effects of reducing the workload of visual inspection and improving the detection accuracy are achieved.
Optionally, the method further comprises:
Acquiring a sample res file;
Analyzing the sample res file, and storing PCB pictures, metadata, a picture imaging algorithm, an AOI detection result and a manual rechecking result contained in the sample res file as files in a preset format;
preprocessing the PCB picture to obtain a sample target picture with a preset size;
defining a sample target picture as a true positive sample, a true negative sample and a false positive sample according to the AOI detection result and the manual re-detection result;
defining the output of the rechecking model as two types of labels; the two types of labels are defect parts and good products respectively;
The defined sample target picture forms training data, the training data is divided into an S domain and a T domain, a true positive sample and a true negative sample form the S domain, and a false positive sample forms the T domain;
Constructing a rechecking model and a loss function corresponding to the rechecking model based on a deep learning algorithm; the rechecking model corresponds to a picture imaging algorithm;
inputting training data into the rechecking model to obtain a trained rechecking model;
Wherein the loss function includes D (TP S,FPT)＜＜D(TPS,TNS),TPS represents a true positive sample in the S domain, FP T represents a false positive sample in the T domain, TN S represents a true negative sample in the S domain, D (-) represents a measure describing distance or a measure describing distribution.
Optionally, a part of the sample target pictures are defined with labels, and in the process of training the review model by using training data, an active learning strategy is adopted to optimize the review model.
Optionally, preprocessing the PCB picture includes:
Removing images outside the AOI detection frame in the PCB picture to obtain an auxiliary picture;
removing PCB wiring information in the auxiliary picture;
The auxiliary picture is adjusted to a predetermined size.
Optionally, the output of the recheck model further includes a result confidence level.
Optionally, the output of the rechecking model further includes a result confidence level and a hotspot graph corresponding to the target picture.
Alternatively, when D (-) represents a measure describing distance, D (-) is used to calculate the maximum mean difference.
Alternatively, when D (-) represents a metric describing the distribution, D (-) is used to calculate the KL divergence.
In a second aspect, an embodiment of the present application provides an automatic optical detection assisting apparatus, including:
The acquisition module is used for acquiring a res file output by the AOI equipment, wherein the res file corresponds to the detection point;
the analysis module is used for solving the res file and storing the PCB picture, the metadata and the picture imaging algorithm contained in the res file into a file in a preset format;
the processing module is used for preprocessing the PCB picture to obtain a target picture with a preset size; the target picture corresponds to a part in the AOI detection frame in the PCB picture;
the rechecking module is used for selecting a rechecking model according to a picture imaging algorithm corresponding to the PCB picture, inputting the target picture into the rechecking model to obtain a rechecking result of the detection point, wherein the rechecking result comprises the target picture and a label; the rechecking model is obtained by training a deep learning algorithm, and the label is a defective part or a good product.
Optionally, the apparatus further comprises a model building module;
The model building module is used for obtaining a sample res file;
Analyzing the sample res file, and storing PCB pictures, metadata, a picture imaging algorithm, an AOI detection result and a manual rechecking result contained in the sample res file as files in a preset format;
preprocessing the PCB picture to obtain a sample target picture with a preset size;
defining a sample target picture as a true positive sample, a true negative sample and a false positive sample according to the AOI detection result and the manual re-detection result;
defining the output of the rechecking model as two types of labels; the two types of labels are defect parts and good products respectively;
The defined sample target picture forms training data, the training data is divided into an S domain and a T domain, a true positive sample and a true negative sample form the S domain, and a false positive sample forms the T domain;
Constructing a rechecking model and a loss function corresponding to the rechecking model based on a deep learning algorithm; the rechecking model corresponds to a picture imaging algorithm;
inputting training data into the rechecking model to obtain a trained rechecking model;
Wherein the loss function includes D (TP S,FPT)＜＜D(TPS,TNS),TPS represents a true positive sample in the S domain, FP T represents a false positive sample in the T domain, TN S represents a true negative sample in the S domain, D (-) represents a measure describing distance or a measure describing distribution.
Optionally, a portion of the sample target pictures are labeled, and the model building module is used for optimizing the review model by adopting an active learning strategy in the process of training the review model by using training data.
Optionally, the processing module is used for removing the image outside the AOI detection frame in the PCB picture to obtain an auxiliary picture;
removing PCB wiring information in the auxiliary picture;
The auxiliary picture is adjusted to a predetermined size.
Optionally, the output of the recheck model further includes a result confidence level.
Optionally, the output of the rechecking model further includes a result confidence level and a hotspot graph corresponding to the target picture.
Alternatively, when D (-) represents a measure describing distance, D (-) is used to calculate the maximum mean difference.
Alternatively, when D (-) represents a metric describing the distribution, D (-) is used to calculate the KL divergence.
In a third aspect, embodiments of the present application provide an automated optical inspection assist apparatus comprising a processor and a memory; the memory has stored therein a program that is loaded and executed by a processor to implement the automated optical inspection assist method as described in the first aspect above.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored therein, the program being loaded and executed by a processor to implement the automatic optical detection assistance method as described in the first aspect above.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present application, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of an automatic optical inspection assistance method according to an embodiment of the present application;
FIG. 2 is a PCB picture corresponding to a certain detection point;
FIG. 3 is a picture obtained during PCB picture preprocessing;
FIG. 4 is a block diagram of an automated optical inspection assist apparatus according to an embodiment of the present application;
fig. 5 is a block diagram of an automatic optical detection assisting device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made more apparent and fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the application are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
In the description of the present application, it should be noted that the directions or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present application, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; the two components can be directly connected or indirectly connected through an intermediate medium, or can be communicated inside the two components, or can be connected wirelessly or in a wired way. The specific meaning of the above terms in the present application will be understood in specific cases by those of ordinary skill in the art.
In addition, the technical features of the different embodiments of the present application described below may be combined with each other as long as they do not collide with each other.
Referring to fig. 1, a flowchart of an automatic optical detection assisting method according to an embodiment of the application is shown, and the method at least includes the following steps:
step 101, acquiring a res file output by the AOI equipment, wherein the res file corresponds to the detection point.
In a PCB packaging production line, AOI equipment performs automatic optical detection according to detection points, obtains AOI detection results aiming at the detection points through an imaging detection algorithm of the AOI equipment, and stores PCB pictures, metadata, picture imaging algorithms, imaging detection algorithms and AOI detection results shot according to a specific picture imaging algorithm into res files corresponding to the detection points.
Optionally, the detection point is a pad, or a portion of an IC pad.
And 102, analyzing the AOI file, and storing the PCB picture, the metadata and the picture imaging algorithm contained in the res file as files in a preset format.
The res file is a binary file, can be checked by using software provided by an AOI equipment manufacturer, but cannot be directly used in a recheck model, so that the res file needs to be analyzed, a PCB picture, metadata and a picture imaging algorithm in the res file are obtained, and the analyzed PCB picture, metadata and picture imaging algorithm are stored as files in a preset format.
The predetermined format is a computer-resolvable format such as: gray level picture, json file.
And step 103, preprocessing the PCB picture to obtain a target picture with a preset size.
The target picture corresponds to a portion of the PCB picture within the AOI detection frame.
And preprocessing the PCB picture obtained by analysis, wherein the PCB picture shot by the AOI equipment comprises detection points and areas except the detection points, the AOI picture is marked with detection frame positions, and an image corresponding to the detection points is in an AOI detection frame.
PCB wiring information is also included on the PCB picture. Through the preprocessing of the PCB picture, the gray level corresponding to the appearance of the element in the PCB picture is reserved, and the wiring information in the PCB picture is removed, so that enough details are reserved in the target picture.
Optionally, the predetermined size is preset.
And 104, selecting a rechecking model according to a picture imaging algorithm corresponding to the PCB picture, and inputting the target picture into the rechecking model to obtain a rechecking result of the detection point, wherein the rechecking result comprises the target picture and the label.
The recheck model is obtained by training a deep learning algorithm in advance, and a corresponding relation exists between a picture imaging algorithm of the AOI equipment and the recheck model. Because the packaging type of the detection point determines a picture imaging algorithm used by the AOI equipment, the packaging type and the picture imaging algorithm have a corresponding relation before, and therefore, the detection points of different packaging types are rechecked by corresponding rechecking models.
Optionally, the image imaging algorithm of the AOI device and the recheck model are in one-to-one correspondence, or the image imaging algorithm of the AOI device and the recheck model are in a mapping relationship of several pairs.
The label is a defective part or a good product; the label is used for indicating whether the PCB product corresponding to the detection point is a good product or a defective product.
After inputting the target picture with the preset size into the rechecking model, outputting the label obtained by the judgment of the model and the target picture based on which the judgment is made by the model by the rechecking model.
Optionally, the output of the recheck model further includes a result confidence level.
In summary, the embodiment of the application provides an automatic optical detection auxiliary method, which comprises the steps of obtaining a res file output by AOI equipment, analyzing the res file, storing a PCB picture, metadata and a picture imaging algorithm contained in the res file as files in a preset format, obtaining a target picture with a preset size before processing the PCB picture, selecting a rechecking model according to the picture imaging algorithm corresponding to the PCB picture, and inputting the target picture into the rechecking model to obtain a rechecking result of a detection point; the problems of high false positive rate, low manual rechecking efficiency and high cost of the detection result of the existing AOI equipment are solved; the effects of reducing the workload of visual inspection and improving the detection accuracy are achieved.
In an alternative embodiment based on the embodiment shown in fig. 1, the output of the rechecking model further includes the result confidence and the hotspot graph corresponding to the target picture, that is, the rechecking result includes the target picture, the label, the result confidence and the hotspot graph corresponding to the target picture.
In one example, different areas on the target picture are marked with different colors, so as to obtain a heat point diagram, and the different colors on the heat point diagram are used for distinguishing the attention degree of the areas, for example: red represents highest attention; the important attention area of the re-inspection model in judgment can be clearly defined through the areas with different colors on the thermal point diagram, so that technicians can understand the re-inspection model, and whether the interpretation standard of the re-inspection model is consistent with the manual experience or not can be judged.
Optionally, the result confidence is displayed on the hotspot graph.
The automated optical inspection assistance method further includes training a review model prior to utilizing the review model to review the inspection points. And training a rechecking model corresponding to the image imaging algorithm aiming at each packaging type, and establishing a corresponding relation between the rechecking model and the image imaging algorithm after the rechecking model is trained.
The training process of the recheck model comprises the following steps:
in step 201, a sample res file is obtained.
The sample res file contains PCB pictures, metadata, a picture imaging algorithm, an AOI detection result and a manual recheck result.
The AOI detection result is a result obtained by detecting by using an imaging detection algorithm in AOI equipment, and the AOI detection result is a good product or a defective product.
Because the false positive rate of the detection result of the AOI equipment is high, when the AOI detection result is a defective part, the detection point is also required to be manually rechecked, the manual rechecked result is input into the AOI equipment after the manual rechecked, and the AOI equipment records and stores the manual rechecked result into the res file.
Optionally, the manual rechecking result is reflected according to the manual rechecking flag bit in the res file.
Step 202, analyzing the sample res file, and storing the PCB picture, metadata, the picture imaging algorithm, the AOI detection result, and the manual review result contained in the sample res file as files in a predetermined format.
The predetermined format is a file format that can be parsed by a computer.
And 203, preprocessing the PCB picture to obtain a sample target picture with a preset size.
And analyzing the sample res file to obtain a PCB picture, preprocessing the PCB picture, reserving gray levels corresponding to the appearance of elements in the PCB picture, and removing routing information in the PCB picture so that enough details are reserved in the sample target picture.
Optionally, the predetermined size is preset.
And 204, defining the sample target picture as a true positive sample, a true negative sample and a false positive sample according to the AOI detection result and the manual recheck result.
When the AOI detection result is a defective part and the manual recheck result is a good product, defining a sample target picture as a false positive sample (FP); when the AOI detection result is a defect part and the manual recheck result is a defect part, defining a sample target picture as true Yang Yangben (TP); and when the AOI detection result is good and the manual recheck result is empty, defining the sample target picture as a true negative sample (TN).
In step 205, the output of the review model is defined as two types of labels.
The two types of labels are defect parts and good products respectively.
The output of each rechecking model obtained through training is two types of labels, namely a defective part and a good product.
Optionally, in order to reduce the workload of manual labeling and improve the training speed of the recheck model, labeling the labels on part of sample target pictures. And determining the label of the sample target picture according to the actual condition of the detection point corresponding to the sample target picture.
And 206, constructing training data by the defined sample target pictures, dividing the training data into an S domain and a T domain, constructing an S domain by the true positive samples and the true negative samples, and constructing the T domain by the false positive samples.
Step 207, constructing a recheck model and a loss function corresponding to the recheck model based on a deep learning algorithm.
The built re-inspection model ensures that the re-inspection model has enough detection rate and extremely small judgment error for true Yang Yangben (TP) and true negative samples (TN), and the loss function comprises D (TP S,FPT)＜＜D(TPS,TNS).
TP S represents a true positive sample in the S domain, FP T represents a false positive sample in the T domain, TN S represents a true negative sample in the S domain, and D (-) represents a measure describing distance or a measure describing distribution.
When D (-) represents a metric describing distance, D (-) is used to calculate MMD (maximum MEAN DISCREPANCY, maximum mean difference). Such as: d (TP S,FPT) was used to calculate the MMD of TP S and FP T.
When D (-) represents a metric describing the distribution, D (-) is used to calculate KL divergence. Such as: d (TP S,FPT) was used to calculate KL for TP S and FP T.
It should be noted that step 207 may also be performed before steps 201 to 205, which is not limited by the embodiment of the present application.
And step 208, inputting the training data into the rechecking model to obtain a trained rechecking model.
Optionally, a part of the sample target pictures are defined with labels, and in the process of training the review model by using training data, an active learning strategy is adopted to optimize the review model.
In the model training process, the model obtained by the previous training is utilized to obtain an output label of a sample target picture of an undefined label in training data, the corresponding result confidence coefficient is output, and a false positive sample with low confidence coefficient is screened out and labels are manually marked by technicians. The labeling noise in the T domain in the training data can be continuously reduced through the active learning strategy.
The method and the loss function for training the reinspection model provided by the embodiment of the application can allow the T domain containing the false positive sample to contain a large amount of marking noise, and simultaneously obtain the reinspection model with enough detection rate and extremely low false positive sample and true negative sample.
Optionally, outputting the hot spot diagram corresponding to the sample target picture while outputting the result confidence. In one example, the resulting confidence is displayed on a heat map, where different colors are used to distinguish between the concerns of different regions.
It should be noted that steps 201 to 208 are performed before step 101.
In the embodiment of the application, the 'preprocessing the PCB picture' can be realized by the following modes:
And step 301, removing the image outside the AOI detection frame in the PCB picture to obtain an auxiliary picture.
In one example, as shown in fig. 2, an AOI inspection frame 21 is displayed on the PCB picture, and the position and size information of the AOI inspection frame is determined according to the configuration of the AOI device; also in the PCB picture is PCB trace information 22.
And removing images outside the AOI detection frame in the PCB picture according to the position and the size information of the AOI detection frame, and removing the AOI detection frame to obtain an auxiliary picture.
When the PCB picture is preprocessed, the AOI detection frame is multiplexed, and meanwhile, the gray level corresponding to the appearance of the element in the PCB picture is reserved as much as possible, so that the target picture obtained after the preprocessing can retain enough details, and defects in the PCB product can be detected conveniently and accurately.
Step 302, removing the PCB routing information in the auxiliary picture.
Because the PCB wiring information can affect defect detection, the PCB wiring information in the auxiliary picture needs to be removed, and the PCB wiring information is completely removed as far as possible.
In one example, the PCB picture is shown in FIG. 2, and the PCB trace information 22 in the AOI detection frame 21 is removed, and the resulting picture is shown in FIG. 3.
Step 303, the auxiliary picture is adjusted to a predetermined size.
The predetermined size is preset. Since the sizes of the AOI inspection frames may be different, the size of the auxiliary picture is adjusted to a predetermined size for the convenience of the review model to make a judgment.
In the process of training the recheck model, the PCB picture contained in the sample res file is processed through steps 301 to 303 to obtain a sample target picture. When the auxiliary AOI equipment is automatically and optically detected by using the re-detection model obtained through training, the PCB picture contained in the res file is processed through steps 301 to 302, and a target picture is obtained.
Optionally, the automatic optical detection auxiliary method provided by the embodiment of the application is applicable to computer equipment connected with the AOI equipment. On a PCB production line, after AOI detection is carried out on the detection points by the AOI equipment, the automatic optical detection auxiliary method provided by the embodiment of the application is directly executed, and the re-detection results detected by the re-detection model are output for res files output by each AOI equipment.
Fig. 4 is a block diagram of an automatic optical detection assisting apparatus according to an embodiment of the present application, where the automatic optical detection assisting apparatus at least includes: the device comprises an acquisition module 410, an analysis module 420, a processing module 430 and a rechecking module 440.
An obtaining module 410, configured to obtain a res file output by the AOI device, where the res file corresponds to the detection point;
The parsing module 420 is configured to parse the res file, and store the PCB picture, the metadata, and the picture imaging algorithm included in the res file as a file in a predetermined format;
The processing module 430 is configured to perform preprocessing on the PCB image to obtain a target image with a predetermined size; the target picture corresponds to a part in the AOI detection frame in the PCB picture;
the rechecking module 440 is configured to select a rechecking model according to a picture imaging algorithm corresponding to the PCB picture, input the target picture into the rechecking model, and obtain a rechecking result of the detection point, where the rechecking result includes the target picture and the label; the rechecking model is obtained by training a deep learning algorithm, and the label is a defective part or a good product.
Optionally, the apparatus further comprises a model building module; the model building module is used for obtaining a sample res file;
Analyzing the sample res file, and storing PCB pictures, metadata, a picture imaging algorithm, an AOI detection result and a manual rechecking result contained in the sample res file as files in a preset format;
preprocessing the PCB picture to obtain a sample target picture with a preset size;
defining a sample target picture as a true positive sample, a true negative sample and a false positive sample according to the AOI detection result and the manual re-detection result;
defining the output of the rechecking model as two types of labels; the two types of labels are defect parts and good products respectively;
The defined sample target picture forms training data, the training data is divided into an S domain and a T domain, a true positive sample and a true negative sample form the S domain, and a false positive sample forms the T domain;
Constructing a rechecking model and a loss function corresponding to the rechecking model based on a deep learning algorithm; the rechecking model corresponds to a picture imaging algorithm;
inputting training data into the rechecking model to obtain a trained rechecking model;
Wherein the loss function includes D (TP S,FPT)＜＜D(TPS,TNS),TPS represents a true positive sample in the S domain, FP T represents a false positive sample in the T domain, TN S represents a true negative sample in the S domain, D (-) represents a measure describing distance or a measure describing distribution.
Optionally, a portion of the sample target pictures are labeled, and the model building module is used for optimizing the review model by adopting an active learning strategy in the process of training the review model by using training data.
Optionally, the processing module is used for removing the image outside the AOI detection frame in the PCB picture to obtain an auxiliary picture;
removing PCB wiring information in the auxiliary picture;
The auxiliary picture is adjusted to a predetermined size.
Optionally, the output of the recheck model further includes a result confidence level.
Optionally, the output of the rechecking model further includes a result confidence level and a hotspot graph corresponding to the target picture.
Alternatively, when D (-) represents a measure describing distance, D (-) is used to calculate the maximum mean difference.
Alternatively, when D (-) represents a metric describing the distribution, D (-) is used to calculate the KL divergence.
For relevant details reference is made to the method embodiments described above.
It should be noted that: in the automatic optical detection auxiliary device provided in the above embodiment, only the division of the above functional modules is used for illustration, and in practical application, the above functional allocation may be performed by different functional modules according to needs, that is, the internal structure of the automatic optical detection auxiliary device is divided into different functional modules to complete all or part of the functions described above. In addition, the automatic optical detection assisting device and the automatic optical detection assisting method provided in the above embodiments belong to the same concept, and detailed implementation processes thereof are shown in the method embodiments, and are not repeated here.
Referring to fig. 5, a block diagram of an automatic optical inspection assisting apparatus according to an exemplary embodiment of the present application is shown. The terminal of the present application may include one or more of the following components: a processor 510 and a memory 520.
Processor 510 may include one or more processing cores. The processor 510 connects various parts within the overall terminal using various interfaces and lines, performs various functions of the terminal and processes data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 520, and invoking data stored in the memory 520. Alternatively, the processor 510 may be implemented in hardware in at least one of digital signal Processing (DIGITAL SIGNAL Processing, DSP), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor 510 may integrate one or a combination of several of a central processing unit (Central Processing Unit, CPU) and a modem, etc. Wherein, the CPU mainly processes an operating system, application programs and the like; the modem is used to handle wireless communications. It will be appreciated that the modem may not be integrated into the processor 510 and may be implemented by a single chip.
Optionally, the automatic optical detection assistance method provided by the above-described method embodiments is implemented when the processor 510 executes the program instructions in the memory 520.
Memory 520 may include random access Memory (Random Access Memory, RAM) or Read-Only Memory (ROM). Optionally, the memory 520 includes a non-transitory computer readable medium (non-transitory computer-readable storage medium). Memory 520 may be used to store instructions, programs, code sets, or instruction sets. The memory 520 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, and the like; the storage data area may store data created according to the use of the terminal, etc.
It should be noted that the above terminal is only illustrative, and the terminal may further include fewer or more components in actual implementation, such as: the device also includes a touch display screen, a communication assembly, a sensor assembly, etc., which is not limited in this embodiment.
It should be noted that, the device performing the steps 101 to 104 and the device performing the steps 201 to 208 are the same device, or the device performing the steps 101 to 104 and the device performing the steps 201 to 208 are different devices; the embodiment of the present application is not limited thereto.
Optionally, the present application further provides a computer readable storage medium having a program stored therein, the program being loaded and executed by a processor to implement the automatic optical detection assistance method of the above-described method embodiment.
Optionally, the present application further provides a computer product, which includes a computer readable storage medium having a program stored therein, the program being loaded and executed by a processor to implement the automatic optical detection-assisted identification method of the above-described method embodiment.
It is apparent that the above examples are given by way of illustration only and are not limiting of the embodiments. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. While still being apparent from variations or modifications that may be made by those skilled in the art are within the scope of the application.

Claims (18)

1. An automated optical inspection assist method, the method comprising:
acquiring a res file output by an AOI device, wherein the res file corresponds to a detection point;
analyzing the res file, and storing PCB pictures, metadata and a picture imaging algorithm contained in the res file as files in a preset format;
Preprocessing the PCB picture to obtain a target picture with a preset size; the target picture corresponds to a part in an AOI detection frame in the PCB picture;
selecting a rechecking model according to a picture imaging algorithm corresponding to the PCB picture, and inputting the target picture into the rechecking model to obtain a rechecking result of the detection point, wherein the rechecking result comprises the target picture and a label;
the method for acquiring the rechecking model comprises the following steps:
Acquiring a sample res file;
Analyzing the sample res file, and storing PCB pictures, metadata, a picture imaging algorithm, an AOI detection result and a manual rechecking result contained in the sample res file as files in a preset format;
preprocessing the PCB picture to obtain a sample target picture with a preset size;
defining the sample target picture as a true positive sample, a true negative sample and a false positive sample according to the AOI detection result and the manual recheck result;
defining the output of the rechecking model as two types of labels; the two types of labels are defect parts and good products respectively;
The defined sample target pictures form training data, the training data is divided into an S domain and a T domain, a true positive sample and a true negative sample form the S domain, and a false positive sample forms the T domain;
constructing a reinspection model and a loss function corresponding to the reinspection model based on a deep learning algorithm; the rechecking model corresponds to a picture imaging algorithm;
and inputting the training data into the rechecking model to obtain a trained rechecking model.
2. The method according to claim 1, characterized in that:
The loss function includes D (TP S,FPT)＜＜D(TPS,TNS),TPS represents a true positive sample in the S domain, FP T represents a false positive sample in the T domain, TN S represents a true negative sample in the S domain, D (-) represents a measure describing distance or a measure describing distribution.
3. The method of claim 2, wherein a portion of sample target pictures are labeled, and wherein the review model is optimized using an active learning strategy during training of the review model using the training data.
4. The method according to claim 1 or 2, wherein the pre-processing the PCB picture comprises:
Removing images outside the AOI detection frame in the PCB picture to obtain an auxiliary picture;
Removing PCB wiring information in the auxiliary picture;
and adjusting the auxiliary picture to a predetermined size.
5. A method according to any one of claims 1 to 3, wherein the output of the review model further comprises a result confidence level.
6. A method according to any one of claims 1 to 3, wherein the output of the review model further comprises a result confidence level and a hotspot graph corresponding to the target picture.
7. The method of claim 2, wherein D (·) is used to calculate the maximum mean difference when D (·) represents a metric describing distance.
8. The method according to claim 2, wherein D (·) is used to calculate the KL divergence when D (·) represents a metric describing the distribution.
9. An automated optical inspection assist device, the device comprising:
the acquisition module is used for acquiring a res file output by the AOI equipment, wherein the res file corresponds to the detection point;
The analysis module is used for resolving the res file and storing the PCB picture, the metadata and the picture imaging algorithm contained in the res file into a file with a preset format;
The processing module is used for preprocessing the PCB picture to obtain a target picture with a preset size; the target picture corresponds to a part in an AOI detection frame in the PCB picture;
The rechecking module is used for selecting a rechecking model according to a picture imaging algorithm corresponding to the PCB picture, inputting the target picture into the rechecking model to obtain a rechecking result of the detection point, wherein the rechecking result comprises the target picture and a label; the rechecking model is obtained by training a deep learning algorithm, and the label is a defective part or a good product;
The model building module is used for obtaining a sample res file;
Analyzing the sample res file, and storing PCB pictures, metadata, a picture imaging algorithm, an AOI detection result and a manual rechecking result contained in the sample res file as files in a preset format;
preprocessing the PCB picture to obtain a sample target picture with a preset size;
defining the sample target picture as a true positive sample, a true negative sample and a false positive sample according to the AOI detection result and the manual recheck result;
defining the output of the rechecking model as two types of labels; the two types of labels are defect parts and good products respectively;
The defined sample target pictures form training data, the training data is divided into an S domain and a T domain, a true positive sample and a true negative sample form the S domain, and a false positive sample forms the T domain;
constructing a reinspection model and a loss function corresponding to the reinspection model based on a deep learning algorithm; the rechecking model corresponds to a picture imaging algorithm;
and inputting the training data into the rechecking model to obtain a trained rechecking model.
10. The apparatus of claim 9, wherein the loss function comprises D (TP S,FPT)＜＜D(TPS,TNS),TPS represents a true positive sample in the S domain, FP T represents a false positive sample in the T domain, TN S represents a true negative sample in the S domain, D (-) represents a measure describing distance or a measure describing distribution.
11. The apparatus of claim 10, wherein the portion of the sample target picture is defined as a label, and wherein the model building module is configured to optimize the review model using an active learning strategy during training of the review model using the training data.
12. The device according to claim 9 or 10, wherein the processing module is configured to remove an image outside the AOI detection frame in the PCB picture to obtain an auxiliary picture;
Removing PCB wiring information in the auxiliary picture;
and adjusting the auxiliary picture to a predetermined size.
13. The apparatus of any one of claims 9 to 11, wherein the output of the review model further comprises a result confidence.
14. The apparatus according to any one of claims 9 to 11, wherein the output of the review model further includes a result confidence level and a hotspot graph corresponding to the target picture.
15. The apparatus of claim 10, wherein D (-) is used to calculate the maximum mean difference when D (-) represents a measure describing distance.
16. The apparatus of claim 10, wherein D (·) is used to calculate the KL divergence when D (·) represents a metric describing the distribution.
17. An automated optical inspection assist apparatus, the apparatus comprising a processor and a memory; stored in the memory is a program that is loaded and executed by the processor to implement the automated optical inspection assist method according to any one of claims 1 to 8.
18. A computer-readable storage medium, in which a program is stored, the program being loaded and executed by a processor to implement the automated optical inspection assist method according to any one of claims 1 to 8.
CN202110707665.6A 2021-03-31 2021-06-24 Automatic optical detection assisting method, device and storage medium Active CN113538341B (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN2021103549913 2021-03-31
CN202110354991 2021-03-31

Publications (2)

Publication Number Publication Date
CN113538341A CN113538341A (en) 2021-10-22
CN113538341B true CN113538341B (en) 2024-04-30

Family

ID=78096743

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110707665.6A Active CN113538341B (en) 2021-03-31 2021-06-24 Automatic optical detection assisting method, device and storage medium

Country Status (1)

Country Link
CN (1) CN113538341B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114066835A (en) * 2021-11-08 2022-02-18 鲁班嫡系机器人（深圳）有限公司 Image detection method, automatic optical detection method, device, storage medium and equipment
CN114265586B (en) * 2021-12-09 2023-12-05 深圳模德宝科技有限公司 Automatic programming method and device for cutting and computer readable storage medium
TWI801038B (en) * 2021-12-16 2023-05-01 新加坡商鴻運科股份有限公司 Defect detection method, system, electronic device and storage medium
CN114895666B (en) * 2022-04-11 2024-04-19 广州镭晨智能装备科技有限公司 Path planning method and device for camera shooting assembly and computer readable storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110455822A (en) * 2019-07-10 2019-11-15 苏州卓融新能源科技有限公司 A kind of detection method of pcb board defect
WO2020143592A1 (en) * 2019-01-07 2020-07-16 鲁班嫡系机器人（深圳）有限公司 Defect recognition model training method and apparatus, computer device and storage medium
CN111768005A (en) * 2020-06-19 2020-10-13 北京百度网讯科技有限公司 Training method and device for lightweight detection model, electronic equipment and storage medium
WO2021027184A1 (en) * 2019-08-09 2021-02-18 苏州康代智能科技股份有限公司 Pcb maintenance system and maintenance method based on false point defect detection

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020143592A1 (en) * 2019-01-07 2020-07-16 鲁班嫡系机器人（深圳）有限公司 Defect recognition model training method and apparatus, computer device and storage medium
CN110455822A (en) * 2019-07-10 2019-11-15 苏州卓融新能源科技有限公司 A kind of detection method of pcb board defect
WO2021027184A1 (en) * 2019-08-09 2021-02-18 苏州康代智能科技股份有限公司 Pcb maintenance system and maintenance method based on false point defect detection
CN111768005A (en) * 2020-06-19 2020-10-13 北京百度网讯科技有限公司 Training method and device for lightweight detection model, electronic equipment and storage medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
基于SSD神经网络的图像自动标注及应用研究;谢禹;李玉俊;董文生;;信息技术与标准化(04);全文 *
液晶显示屏背光源模组表面缺陷自动光学检测***设计;史艳琼;卢荣胜;张腾达;;传感技术学报(05);全文 *

Also Published As

Publication number Publication date
CN113538341A (en) 2021-10-22

Similar Documents

Publication Publication Date Title
CN113538341B (en) Automatic optical detection assisting method, device and storage medium
JP5522065B2 (en) Board inspection system
US10674651B2 (en) Printed circuit board inspecting apparatus, method for determining fault type of screen printer and computer readable recording medium
US7672501B2 (en) Substrate inspection system including a visual inspection device for inspection by image processing
US7869644B2 (en) Methods of and apparatus for inspecting substrate
JP5948797B2 (en) System, apparatus and method for supporting visual confirmation of inspection results
US7406191B2 (en) Inspection data producing method and board inspection apparatus using the method
CN110490181B (en) Form filling and auditing method, device and equipment based on OCR (optical character recognition) technology and computer storage medium
JP5861462B2 (en) Inspection standard registration method for solder inspection and board inspection apparatus using the method
JP2007003297A (en) Inspection device of substrate, parameter setting method thereof and parameter setting device
JP3906780B2 (en) Data registration method for component code conversion table, board inspection data creation device, registration processing program, and storage medium thereof
JP4539355B2 (en) Substrate inspection device, parameter setting method and parameter setting device
CN113222913B (en) Circuit board defect detection positioning method, device and storage medium
JP2004340832A (en) Method and system for visual inspection of circuit board
CN113030121A (en) Automatic optical detection method, system and equipment for circuit board components
CN112014404A (en) Component detection method, device, system, electronic equipment and storage medium
CN113506243A (en) PCB welding defect detection method and device and storage medium
CN116337868A (en) Surface defect detection method and detection system
KR102228957B1 (en) Printed circuit board inspecting apparatus, method for determining fault type of screen printer and computer readable recording medium
CN113077416A (en) Welding spot welding defect detection method and system based on image processing
CN113724218A (en) Method and device for identifying chip welding defects by image and storage medium
CN117589770A (en) PCB patch board detection method, device, equipment and medium
CN112446865A (en) Flaw identification method, flaw identification device, flaw identification equipment and storage medium
JP2008241424A (en) Image processing apparatus and method using the same
CN115937555A (en) Industrial defect detection algorithm based on standardized flow model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant