CN114881998A

CN114881998A - Workpiece surface defect detection method and system based on deep learning

Info

Publication number: CN114881998A
Application number: CN202210587862.3A
Authority: CN
Inventors: 韩小平; 王立华; 闫云昊; 窦永旺; 赫金娜; 杨朋达
Original assignee: Qinhuangdao Weikawei Faurecia Automotive Interior Parts Co ltd
Current assignee: Qinhuangdao Weikawei Faurecia Automotive Interior Parts Co ltd
Priority date: 2022-05-26
Filing date: 2022-05-26
Publication date: 2022-08-09

Abstract

The invention discloses a workpiece surface defect detection method and system based on deep learning. The method comprises the steps of carrying out defect detection on a product of a client by using an image-based deep learning algorithm, focusing the target detection algorithm on the defect, establishing a sample distribution data set about each defect of an aluminum plate by using the morphological outline and the characteristics of the defect in a two-dimensional image to have a certain rule and range, researching the characteristic distribution of the sample sets on the two-dimensional image by using a neural network algorithm, establishing a mathematical model, inputting the model into a product image acquired by the client on site, outputting the coordinates of points which are possibly the defects in the image by using model calculation, and judging whether the product is defective or not by screening the result.

Description

Workpiece surface defect detection method and system based on deep learning

Technical Field

The invention relates to the technical field of defect detection, in particular to a workpiece surface defect detection method and system based on deep learning.

Background

With the rapid development of the industry, people pay more and more attention to the quality of products, and the requirements are more and more strict. Because some defects often appear in the production process of the product, and the defects have certain randomness, namely the defect types and the shapes and the sizes are different, the detection of the surface defects of the product is one of the most important processes in the production process, and the detection directly influences the product quality and the user experience.

At present, the defects of most industrial parts are mostly detected by adopting a manual visual quality inspection mode. However, this method has the following drawbacks: the efficiency is low: the efficiency of checking parts actually tests the proficiency of one person, the higher detection efficiency can be realized if the working time is longer, but the personal fatigue and lazy inertia can be increased along with the increase of the working time, so that the detection efficiency of quality testing personnel can be reduced; risk of missed detection: as the working time increases, the personal attention also decreases, the risk of missed detection is brought, the machine is not tired, and the problem does not exist; hard to define: because the sizes of industrial defects are all in millimeter level, the defects in millimeter level are difficult to be distinguished manually by naked eyes; the quantitative analysis is difficult: the manual defect judgment cannot carry out data statistics, and the factory intelligence is hindered; the labor cost is high.

Later, surface defect detection technology based on deep learning was proposed, which has a core that machine vision is used to participate in quality monitoring of products instead of human vision. The technology can be rapidly applied to the defect detection procedure of the product surface from the date of the proposal, and becomes the development trend of the current manufacturing industry because the labor cost can be reduced to eliminate the influence of the judgment difference caused by the subjective consciousness and the visual fatigue of people, the detection efficiency and the detection precision can be improved, and the detection error is reduced.

However, the existing deep learning algorithm has weak detection capability, slow operation speed, easy processing of deep learning problems such as existence of simple detection, defects of regular detection number and shape and size, and the like, but has low accuracy and speed when detecting defects of complex surfaces in the field of industrial quality inspection.

Disclosure of Invention

The invention provides a workpiece surface defect detection method based on deep learning.

The invention provides the following scheme:

a workpiece surface defect detection method based on deep learning comprises the following steps:

acquiring sample pictures of various defect types included in a product to generate a training sample set, wherein the defect types at least comprise scratches, abrasions, bumps, pits, white dots and watermarks;

inputting the training sample set into a surface defect detection model based on a deep neural network, and training the surface defect detection model based on the deep neural network to obtain a surface defect detection model;

acquiring a surface picture of a product to be detected, and inputting the surface picture into the surface defect detection model;

and obtaining a detection result output by the surface defect detection model, and determining whether the surface of the product to be detected contains defects or not after threshold value screening, size screening and area screening are sequentially carried out on the detection result.

Preferably: the obtaining of the sample pictures of the defect types included in the product includes:

and respectively acquiring a target number of sample pictures of each defect type, wherein the target number is the set minimum number of samples required by meeting the training of the surface defect detection model.

Preferably: the respectively obtaining of the sample pictures of the target number of each defect type includes:

the acquiring mode of the sample picture comprises acquiring a real defect picture of the defect type and/or an intelligent simulation defect picture of the defect type and/or an artificial manufacture defect picture of the defect type.

Preferably: the training sample set comprises an image matrix, and the matrix comprises a four-dimensional array; and cutting according to the size of the image, and inputting the cut image into a surface defect detection model based on the deep neural network in batches.

Preferably: the surface defect detection model includes convolution layers that employ a depth convolution algorithm.

Preferably: the detection result output by the surface defect detection model comprises a plurality of multi-scale prediction results.

Preferably: the multi-scale prediction result comprises the size of the output feature map as y 1: (13 × 13), y 2: (26 × 26), y 3: (52 × 52).

Preferably: the surface defect detection model receives one (416 × 416) surface picture, and performs downsampling through convolution with 5 step sizes of 2 to enable y1 to be output (13 × 13); upsampling from the convolutional layer of the penultimate layer of y1 is connected to the last eigenmap tensor of size 26 × 26 to make y2 output (26 × 26); upsampling from the convolutional layer of the penultimate layer of y2 is connected to the last eigenmap tensor of size 52 x 52 to make y3 output (52 x 52).

Preferably: the threshold screening comprises the steps of obtaining detection results output by the surface defect detection model and the confidence coefficient of each detection result, wherein the confidence coefficient is a decimal number between 0 and 1; removing the detection result with the confidence coefficient lower than a confidence coefficient threshold value;

the size screening comprises setting a size threshold to be compared with the confidence of the detection frame, and detecting the size of the detection frame is larger than the size of the detection frame;

the area screening comprises the step of manufacturing a mask matrix of the area screening according to the process reference of the product to be detected, wherein the mask matrix is a two-dimensional matrix of 0 and 1, and the length and the width are the length and the width of the surface picture; and judging whether the frame is in the area to be detected according to whether the coordinate of the detection frame is 0 or 1 in the mask matrix.

A deep learning based workpiece surface defect detection system, the system comprising:

the training sample set acquisition unit is used for acquiring sample pictures of various defect types included in a product to generate a training sample set, wherein the defect types at least comprise scratches, abrasions, knocks, pits, white dots and watermarks;

the surface defect detection model determining unit is used for inputting the training sample set into a surface defect detection model based on a deep neural network, and training the surface defect detection model based on the deep neural network to obtain a surface defect detection model;

the surface picture input unit is used for acquiring a surface picture of a product to be detected and inputting the surface picture into the surface defect detection model;

and the data post-processing unit is used for acquiring the detection result output by the surface defect detection model, and determining whether the surface of the product to be detected contains defects or not after threshold screening, size screening and area screening are sequentially carried out on the detection result.

According to the specific embodiment provided by the invention, the invention discloses the following technical effects:

according to the invention, a workpiece surface defect detection method based on deep learning can be realized, and in an implementation mode, the method can comprise the steps of obtaining sample pictures of various defect types included in a product to generate a training sample set, wherein the defect types at least comprise scratches, abrasions, scratches, bumps, pits, white dots and watermarks; inputting the training sample set into a surface defect detection model based on a deep neural network, and training the surface defect detection model based on the deep neural network to obtain a surface defect detection model; acquiring a surface picture of a product to be detected, and inputting the surface picture into the surface defect detection model; and obtaining a detection result output by the surface defect detection model, and determining whether the surface of the product to be detected contains defects or not after threshold value screening, size screening and area screening are sequentially carried out on the detection result. The method for detecting the surface defects of the workpiece based on the deep learning comprises the steps of detecting the defects of a product of a client by using the deep learning algorithm based on an image, focusing the target detection algorithm on the defects, collecting a certain number of defect images, establishing a sample distribution data set related to each defect of an aluminum plate, supporting the data sets by the deep learning, researching the characteristic distribution of the sample sets on the two-dimensional image through a neural network algorithm, establishing a mathematical model, inputting the model into a product image acquired on site by the client, calculating the model based on the defect learning through the deep learning algorithm, the coordinates of the points which are possibly defective in the image can be output, and whether the product is defective or not can be judged by screening the result.

Of course, it is not necessary for any product in which the invention is practiced to achieve all of the above-described advantages at the same time.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.

FIG. 1 is a flowchart of a method for detecting surface defects of a workpiece based on deep learning according to an embodiment of the present invention;

FIG. 2 is a network structure diagram based on deep learning algorithm according to an embodiment of the present invention;

FIG. 3 is a diagram of network results provided by embodiments of the present invention;

FIG. 4 is a schematic diagram of a head portion of a network provided by an embodiment of the present invention;

fig. 5 is a schematic diagram of a workpiece surface defect detection system based on deep learning according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived from the embodiments of the present invention by a person skilled in the art, are within the scope of the present invention.

Examples

Referring to fig. 1, a method for detecting surface defects of a workpiece based on deep learning according to an embodiment of the present invention is shown in fig. 1, and the method may include:

s101: acquiring sample pictures of various defect types included in a product to generate a training sample set, wherein the defect types at least comprise scratches, abrasions, bumps, pits, white dots and watermarks;

s102: inputting the training sample set into a surface defect detection model based on a deep neural network, and training the surface defect detection model based on the deep neural network to obtain a surface defect detection model;

s103: acquiring a surface picture of a product to be detected, and inputting the surface picture into the surface defect detection model;

s104: and obtaining a detection result output by the surface defect detection model, and determining whether the surface of the product to be detected contains defects or not after threshold value screening, size screening and area screening are sequentially carried out on the detection result.

The workpiece surface defect detection method based on deep learning provided by the embodiment of the application can be used for respectively obtaining sample pictures of each type of defects on the surface of a product to generate a training sample set aiming at the various types of defects on the surface of the product. Compared with the traditional mode of adopting a non-defect sample as a training sample set, the method provided by the application can be used for more accurately judging various types of defects. The purpose of further improving the defect detection precision is achieved.

Specifically, the obtaining of the sample pictures of each defect type included in the product includes:

and respectively acquiring sample pictures of target quantity of each defect type, wherein the target quantity is the set minimum sample quantity required by meeting the training of the surface defect detection model.

In order to ensure that the number of samples can meet the requirement of model training, when the number of samples of each defect type is collected, the number of samples needs to be ensured to reach a certain number requirement. However, when a defect of a sample is actually collected, it may occur that a defect type of the actual product is less in the sample, which makes the collection difficult. In order to solve this problem, an embodiment of the present application may further provide that the obtaining of the sample picture includes obtaining an actual defect picture of the defect type and/or an intelligent simulated defect picture of the defect type and/or an artificial defect picture of the defect type.

In the method provided by the embodiment of the application, a real defect picture can be obtained for each defect type, and when the number of the real defect pictures is small, a part of the pictures of the defect type can be generated in an intelligent simulation mode. The method can also be used for artificially manufacturing defects on the surface of an experimental product, wherein the defects are opposite to the defect types, such as various scratch defects and the like. After the surface of the product is endowed with the defect, the defect part can be photographed and used as a sample picture.

Specifically, the training sample set includes an image matrix, and the matrix includes a four-dimensional array; and cutting according to the size of the image, and inputting the cut image into a surface defect detection model based on the deep neural network in batches.

The surface defect detection model includes convolution layers that employ a depth convolution algorithm. The detection result output by the surface defect detection model comprises a plurality of multi-scale prediction results. The multi-scale prediction result comprises the size of the output feature map as y 1: (13 × 13), y 2: (26 × 26), y 3: (52 × 52). The surface defect detection model receives one (416 × 416) surface picture, and performs down-sampling through convolution with 5 steps of 2 to output y1 (13 × 13); upsampling from the convolutional layer of the penultimate layer of y1 is connected to the last eigenmap tensor of size 26 × 26 to make y2 output (26 × 26); upsampling from the convolutional layer of the penultimate layer of y2 is connected to the last eigenmap tensor of size 52 x 52 to make y3 output (52 x 52).

Whether the product has defects can be judged by screening the results, and the accuracy of determining the defects is ensured. The threshold screening comprises the steps of obtaining detection results output by the surface defect detection model and the confidence coefficient of each detection result, wherein the confidence coefficient is a decimal number between 0 and 1; removing the detection result with the confidence coefficient lower than a confidence coefficient threshold value;

The method for detecting the surface defects of the workpiece based on the deep learning provided by the embodiment of the application is described in detail below. The target detection work flow is as follows: collecting defect samples, deep learning neural network training, model deployment online and model iteration.

Data input: and receiving an image matrix from a software end, wherein the matrix is a four-dimensional array, and at the moment, the image matrix can be cut according to the size of the image and is sent to the next stage in batches after being divided.

And (3) deep convolution algorithm: the high-dimensional image matrix enters a convolutional neural network for feature extraction, aiming at the defect characteristics in the quality inspection field, the algorithm in the technology carries out targeted matching on small defects such as scratches, collisions and pits, watermarks, large-range pattern deformation and the like, so that the output of the neural network has strong size variability. The algorithm adopted in the technology is based on single-stage target detection.

The network structure is as follows:

referring to fig. 2, the DBL is a basic component of the network composition. The Conv convolution layer is followed by BatchNormalization (BN) and LeakyReLU. Except for the last convolution layer, in the network near head part, BN and LeakyReLU form the inseparable part of the convolution layer, and together form the minimum component.

A 5 res structure is used in the backbone network. n represents a number, including res1, res2, …, res8, etc., which means that this res _ block contains n res _ units, which is a large component of the network. Based on the residual structure of ResNet, the use of this structure allows deeper network structures. To ensure better detection accuracy.

There is a tensor concatenation (concat) operation on the prediction branch. The realization method is to splice the middle layer and the up-sampling of a layer behind the middle layer. It is worth noting that the operation of tensor splicing and add of Res _ unit structure is not the same, tensor splicing expands the dimensionality of the tensor, and adding add directly alone does not result in a change in the tensor dimensionality.

The total network depth is set to a total of 252 levels. 23 Res _ units correspond to 23 add layers. The numbers of BN layers and LeakyReLU layers are both 72 layers, and the network structure shows that: each layer BN is followed by a layer of LeakyReLU. The upsampling and tensor stitching operations are 2 each, and 5 zero padding correspond to 5 res _ blocks. The convolutional layer had 75 layers, 72 of which were followed by DBL consisting of BatchNormalization and LeakyReLU. The outputs of three different scales correspond to three convolutional layers, the number of convolutional cores of the last convolutional layer is 255, and for 80 classes of the COCO data set: 3 × (80+4+1) ═ 255, 3 indicates that one grid cell contains 3 bounding boxes, 4 indicates 4 coordinate information of the frame, and 1 indicates confidence.

Fig. 3 is a diagram of a specific network result.

In order to accurately locate and classify targets, the head portion of the design network is shown in FIG. 4.

Regardless of the details of the neural network structure, in general, for an input image, the output tensor, which is mapped to 3 scales, represents the probability that various objects exist at various positions of the image.

For one 416 × 416 input image, 3 a priori boxes are placed per grid of feature maps at each scale, for a total of 13 × 3+26 × 3+52 × 3 — 10647 predictions. Each prediction is an (4+1+80) ═ 85 dimensional vector, which 85 dimensional vector contains the bounding box coordinates (4 values), the bounding box confidence (1 value), the probability of the object class (e.g., there are 80 objects for the standard coco dataset for defect data experiments).

Multi-scale prediction:

to solve the multi-scale prediction, it can be seen in the above network structure diagram that it is assumed that 3 boxes are predicted per grid cell, so each box needs five basic parameters (x, y, w, h, confidence). The backbone outputs feature maps of 3 different scales, y1, y2, y3 as shown in the above figure. The depth of y1, y2 and y3 is 255 and the side length is 13:26: 52.

The feature size obtained for each prediction task is N × [3 × (4+1+80) ], N is the grid size, 3 is the number of bounding boxes obtained per grid, 4 is the number of bounding box coordinates, 1 is the target prediction value, and 80 is the number of categories. For the standard dataset COCO categories, there are 80 categories of probabilities, so each box should output one probability for each category. So 3 × (5+80) ═ 255. This is 255. For actual defect detection items, the category number needs to be determined according to actual conditions

The model uses an up-sampling method to realize the multi-scale feature map. On the basis of the obtained feature map, a first feature map is obtained through six DBL structures and the last convolutional layer in the Darknet-53 part of the backbone network, and first prediction is carried out on the feature map.

And on the Y1 branch, the output of the last 3 rd convolutional layer from back to front is subjected to one DBL structure and one (2,2) upsampling, the upsampling characteristics are connected with the convolutional characteristic tensor output by the 2 nd Res8 structure, a second characteristic map is obtained through six DBL structures and the last convolutional layer, and second prediction is carried out on the characteristic map.

And on the Y2 branch, the output of the 3 rd convolutional layer from back to front is subjected to one DBL structure and one (2,2) upsampling, the upsampling characteristics are connected with the convolutional characteristic tensor output by the 1 st Res8 structure, a third characteristic map is obtained through six DBL structures and the last convolutional layer, and third prediction is carried out on the characteristic map.

The feature map size of the multi-scale prediction output is y 1: (13 × 13), y 2: (26 × 26), y 3: (52 × 52). The network receives one (416 × 416) graph, down-samples by 5 convolutions with step size 2 (416/2^5 ^ 13, y1 output (13 × 13); up-samples from the penultimate convolutional layer of y1 (x2, up-sampling) are then connected to the last feature map tensor of 26 × 26 size, y2 output (26 × 26); up-samples from the penultimate convolutional layer of y2 (x2, up-sampling) are then connected to the last feature map tensor of 52 × 52 size, and y3 output (52 × 52).

And (3) data post-processing: the exported data is processed by three post-processing algorithms:

the first is threshold screening, the network outputs a target detection result box and a confidence coefficient x thereof, the confidence coefficient x is a decimal number between 0 and 1, and the higher the confidence coefficient is, the higher the algorithm is, the more confident the algorithm is that the target is positioned. Most of the low confidence box can be removed by threshold screening.

The second is size screening, which similarly sets a size threshold against the confidence of the detected box, and larger sizes can be detected.

And the third is area screening, wherein the position of the object to be detected in the image is fixed during detection, such as a certain surface of an engine shell, so that a mask matrix for area screening can be manufactured according to the process standard of the object to be detected, the matrix is a two-dimensional matrix of 0 and 1, and the length and the width are the length and the width of the image. Whether the frame is in the area to be detected can be judged according to whether the coordinate of the detection frame is 0 or 1 in the mask matrix.

Through actual measurement, the method provided by the embodiment of the application is adopted to detect the recall rate of external defects (black spots and scratches) with the diameter larger than 0.3mm and processing defects (missing printing and bump deformation) on the aluminum plate by 97 percent, and the precision is 95 percent.

In summary, the method for detecting the surface defect of the workpiece based on the deep learning provided by the application has the advantages that the defect target detection uses the deep learning algorithm based on the image to detect the defect of the product of a client, the target detection algorithm focuses on the defect, general defects such as scratch, abrasion, bump, pit, white point, watermark and the like, the morphological outline and the characteristics of the defects have a certain rule and range on a two-dimensional image, when a sample set is enough, namely a certain number of defect images are collected, a sample distribution data set related to each defect of an aluminum plate can be established, the deep learning uses the data sets as a support, the characteristic distribution of the sample sets on the two-dimensional image is researched through a neural network algorithm, a mathematical model can be established, the input of the model is the product image collected by the client on site, the basis of the model calculation learned by the deep learning algorithm based on the defect, the coordinates of the points which are possibly defective in the image can be output, and whether the product is defective or not can be judged by screening the result.

Referring to fig. 5, corresponding to the method for detecting surface defects of a workpiece based on deep learning provided by the embodiment of the present application, as shown in fig. 5, an embodiment of the present application further provides a system for detecting surface defects of a workpiece based on deep learning, where the system specifically includes:

a training sample set obtaining unit 201, configured to obtain sample pictures of various defect types included in a product to generate a training sample set, where the defect types at least include scratches, abrasions, bumps, pits, white dots, and watermarks;

the surface defect detection model determining unit 202 is configured to input the training sample set into a surface defect detection model based on a deep neural network, and train the surface defect detection model based on the deep neural network to obtain a surface defect detection model;

the surface picture input unit 203 is used for acquiring a surface picture of a product to be detected and inputting the surface picture into the surface defect detection model;

and the data post-processing unit 204 is configured to obtain a detection result output by the surface defect detection model, and determine whether the surface of the product to be detected contains a defect after performing threshold screening, size screening and area screening on the detection result in sequence.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. A workpiece surface defect detection method based on deep learning is characterized by comprising the following steps:

2. The method for detecting the surface defects of the workpiece based on the deep learning as claimed in claim 1, wherein the step of obtaining sample pictures of each defect type included in the product comprises the following steps:

3. The method for detecting the surface defects of the workpiece based on the deep learning as claimed in claim 2, wherein the step of respectively acquiring the target number of sample pictures of each defect type comprises the following steps:

4. The method of claim 1, wherein the training sample set comprises an image matrix, the matrix comprising a four-dimensional array; and cutting according to the size of the image, and inputting the cut image into a surface defect detection model based on the deep neural network in batches.

5. The method of claim 1, wherein the surface defect inspection model comprises convolution layers using a deep convolution algorithm.

6. The method for detecting the surface defects of the workpiece based on the deep learning as claimed in claim 5, wherein the detection results output by the surface defect detection model comprise a plurality of multi-scale prediction results.

7. The method of claim 6, wherein the multi-scale prediction result comprises an output feature map with a size y 1: (13 × 13), y 2: (26 × 26), y 3: (52 × 52).

8. The method of claim 7, wherein the surface defect detection model receives a (416 x 416) picture of the surface, and performs downsampling by 5 convolutions with step size 2 to output y1 as (13 x 13); upsampling from the convolutional layer of the penultimate layer of y1 is connected to the last eigenmap tensor of size 26 × 26 to make y2 output (26 × 26); upsampling from the convolutional layer of the penultimate layer of y2 is connected to the last eigenmap tensor of size 52 x 52 to make y3 output (52 x 52).

9. The method for detecting the surface defects of the workpiece based on the deep learning as claimed in claim 1, wherein the threshold screening comprises obtaining the detection results output by the surface defect detection model and the confidence coefficient of each detection result, wherein the confidence coefficient is a decimal number between 0 and 1; removing the detection result with the confidence coefficient lower than a confidence coefficient threshold value;

the size screening comprises setting a size threshold to be compared with the confidence of the detection frame, and detecting that the size is larger than the size;

10. A workpiece surface defect detection system based on deep learning, the system comprising: