CN111161243A - Industrial product surface defect detection method based on sample enhancement - Google Patents

Industrial product surface defect detection method based on sample enhancement Download PDF

Info

Publication number
CN111161243A
CN111161243A CN201911390407.9A CN201911390407A CN111161243A CN 111161243 A CN111161243 A CN 111161243A CN 201911390407 A CN201911390407 A CN 201911390407A CN 111161243 A CN111161243 A CN 111161243A
Authority
CN
China
Prior art keywords
picture
defect
new
rectangular
detection
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911390407.9A
Other languages
Chinese (zh)
Other versions
CN111161243B (en
Inventor
许玉格
郭子兴
戴诗陆
吴宗泽
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China University of Technology SCUT
Original Assignee
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China University of Technology SCUT filed Critical South China University of Technology SCUT
Priority to CN201911390407.9A priority Critical patent/CN111161243B/en
Publication of CN111161243A publication Critical patent/CN111161243A/en
Application granted granted Critical
Publication of CN111161243B publication Critical patent/CN111161243B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Quality & Reliability (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a sample enhancement based industrial product surface defect detection method, which comprises the following steps: 1) standardizing and normalizing the size of the surface image of the industrial product, cutting and classifying; 2) carrying out data enhancement of random turning on the pictures with the defects; 3) randomly splicing and enhancing the pictures with the defects and the normal pictures; 4) performing iterative training by using a Cascade-RCNN algorithm; 5) obtaining a Cascade-RCNN detection model; 6) and performing sliding window detection on the surface picture of the industrial product to be detected and the texture template picture determined to be free of defects through a Cascade-RCNN detection model, splicing the results detected by the sliding window, comparing the results obtained by the sliding window and the texture template picture, and finally obtaining the defect type and the region label of the picture to be detected. The invention can effectively reduce the influence of conditions such as illumination, exposure and displacement on defect detection, improves the detection stability, improves the resolution capability of the two-stage target detector on patterns and backgrounds, and reduces the false detection rate.

Description

Industrial product surface defect detection method based on sample enhancement
Technical Field
The invention relates to the technical field of industrial product surface defect detection, in particular to a sample enhancement-based industrial product surface defect detection method.
Background
Defect detection is an important part of the production process, and ensures the reliability of industrial products. Surface defect detection of industrial products requires precise positioning of defect positions on a surface and classification of the positioned defects, which is a typical target detection problem. In the past, the surface defect detection technology of industrial products generally uses the traditional machine vision technology to perform operations such as picture gray level binarization, edge contour extraction, template matching and the like, and the defects of the operations are that the operations are very sensitive to changes such as illumination, displacement and the like of pictures and the robustness is poor. In addition, previous surface defect detection studies were based on solid color product surfaces, and since the surface texture features of specially textured products and the defect texture features are similar, it is difficult to distinguish between the two in previous methods.
The target detection in deep learning is realized by taking a convolutional neural network as a feature extractor, and the extracted feature graph is insensitive to changes such as illumination, displacement and the like and has better robustness. A two-stage target detector is composed of a Region Proposal Network (Region Proposal Network) and a classification regression Network, wherein the Region Proposal Network is responsible for generating suggestions of regions where targets may be located, and the classification regression Network classifies the suggested regions and finely adjusts a labeling frame. The function of the network consists of classification loss and regression loss weighting, and a random gradient descent method is adopted for back propagation iteration.
The existing two-stage deep learning target detector has high precision and good universality, but the problems that defects and background textures are difficult to distinguish, normal pictures without the defects cannot participate in model training, pictures of industrial products have high video memory requirements and the like still exist in the surface defect detection with the textures.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, provides a sample enhancement-based industrial product surface defect detection method, can effectively reduce the influence of conditions such as illumination, exposure and displacement on defect detection, improves the detection stability, improves the resolution of a two-stage target detector on patterns and backgrounds, and reduces the false detection rate.
In order to achieve the purpose, the technical scheme provided by the invention is as follows: the industrial product surface defect detection method based on sample enhancement comprises the following steps:
1) carrying out size standardization operation on a picture set on the surface of an industrial product, wherein the picture containing the defects has a corresponding defect marking file, carrying out cutting operation on the defect picture and the defect marking file corresponding to each picture, and dividing the defect picture and the defect marking file into a normal picture set and a defect picture set according to the cut marks;
2) normalizing and enhancing online random data of the defect picture set obtained in the step 1), including randomly turning the defect picture set up and down, left and right, and dividing the defect picture set into batches;
3) for each defective picture in the batch in the step 2), randomly searching a normal picture which corresponds to the defective picture and has the same cutting position and the same texture template pattern in the normal picture set, performing left or right splicing operation on the normal picture and the defective picture, and correspondingly modifying the label file;
4) performing iterative training on the pictures and labels of each batch obtained in the step 3) by using a Cascade-RCNN algorithm, and finishing a round of training after finishing training all batches;
5) after finishing one round of training, repeating the steps 2) to 4) until reaching the set iteration round, outputting and storing parameters in the network to obtain a Cascade-RCNN detection model;
6) and 5) performing sliding window detection on the surface picture of the industrial product to be detected and the determined non-defective texture template picture by using the Cascade-RCNN detection model obtained in the step 5), splicing the results detected by the sliding window, and comparing the results obtained by the sliding window and the determined non-defective texture template picture to obtain the defect type and the region label of the picture to be detected finally.
In step 1), the image set on the surface of the industrial product comprises a template image set Z consisting of a defect-containing image set X, a defect-free normal image set Y and a pattern example image of each texture template, wherein the defect image set X contains labels, and each defect label is a rectangular label frame with a format of (name, category, X)min,ymin,xmax,ymax) Where name represents the picture name, category represents the type of defect, (x)min,ymin) The horizontal and vertical coordinates (x) of the upper left corner of the rectangular labeling boxmax,ymax) The horizontal and vertical coordinates of the lower right corner of the rectangular labeling frame are represented, and neither the picture set Y nor the picture set Z has labeling information; carrying out size standardization on the three picture sets to enable all pictures to be RGB pictures of H x W, wherein H and W are the height and width of the pictures;
the same average cutting is carried out on the three picture sets, and the defect marks on the picture set X are cut according to the following rules: cutting the defective rectangular marking frame in the same way as the picture, mapping the defective rectangular marking frame into the range of the cut small picture, if the marking is cut off, calculating the proportion of the area of the cut rectangular marking frame to the area of the original rectangular marking frame, if the area is larger than a set threshold epsilon, keeping the marking, and otherwise, discarding;
dividing the picture into a defect picture set X according to the cut marking informationnewSet of normal pictures YnewAnd template picture set ZnewAnd stored according to the position where they were cut.
The step 3) comprises the following steps:
3.1) for the defect picture x which is randomly flipped in step 2)i∈XnewFinding out corresponding normal picture set M e (Y) according to template texture and cutting positionnew,Znew) So that x isiAnd the texture template and cut position are the same for all samples in M, where XnewFor a set of cut defect pictures, Ynew,ZnewRespectively a normal picture set and a template picture set after cutting;
3.2) randomly selecting a normal picture yiE.g. M, according to xiSize of (a) to yiIs filled with a value of 0, such that yiSize and x ofiAre all the same in size and for yiCarrying out normalization processing;
3.3) to yiCarrying out data enhancement operation, namely randomly turning up, down, left and right;
3.4) generating a random number between (0,1) at 50%Probability is such that xnew＝(xi,yi) Another 50% probability is such that xnew＝(yi,xi) I.e. randomly splicing left or right, where xnewRepresenting the generated new sample;
3.5) processing the marking information according to the splicing mode, if the picture x has defectsiOn the left, there is no need to change the defect label, if the defect picture xiAnd on the right, the rectangular marking box needs to be corrected correspondingly.
In step 4), the Cascade-RCNN algorithm comprises a trunk network, a region proposing network and a classification regression network, which are respectively used for extracting features, generating region suggestions and classifying and fine-tuning candidate frames; the convolutional neural network ResNeXt-101 and the feature pyramid FPN are used as a backbone network, the area proposal network uses an area proposal network part in a two-stage target detector, namely, fast-RCNN, and the classification regression network uses a multilayer cascade network.
In step 6), the following detection process is performed:
6.1) for an industrial product surface picture to be detected, using a preset sliding window size, using the Cascade-RCNN detection model obtained in the step 5) to perform sliding window detection on the picture to be detected, and mapping the result back to the area of the original image to obtain the label format (category, x) of each defectmin,ymin,xmax,ymaxScore), category denotes the defect class, (x)min,ymin) The horizontal and vertical coordinates (x) of the upper left corner of the rectangular labeling boxmax,ymax) The horizontal and vertical coordinates of the lower right corner of the rectangular labeling frame are represented, score represents the confidence coefficient of defect judgment, and the value of the confidence coefficient is (0, 1);
6.2) judging the defect labels close to the edges of the sliding windows, if the adjacent sliding windows have labels with the same category and the similar position size, carrying out label merging operation according to the confidence degree sequence, wherein the merged rectangular label frame is the minimum circumscribed rectangle of a plurality of rectangular label frames, and calculating the new confidence degree to obtain the average value of the minimum circumscribed rectangle, which is as follows:
scorenew＝(score1+score2+...+scoren)/n
wherein, scorenewIndicates the new confidence, scoreiRepresenting the confidence of the ith rectangular labeling frame participating in synthesis, and n representing the total number of the rectangular labeling frames participating in synthesis;
6.3) adopting the steps 6.1) and 6.2) to carry out defect detection on the template picture set Z in advance under the offline condition, and storing the obtained result;
6.4) detecting the surface picture of the industrial product to be detected online by adopting the steps 6.1) and 6.2), comparing the obtained result with the detection result of the corresponding template stored in the step 6.3), adopting IoU as a comparison standard, and obtaining the following calculation formula:
Figure BDA0002344783280000051
wherein DR represents a defect rectangular marking frame detected on the to-be-detected image, and GT represents a real defect rectangular marking frame; the specific method for comparison is as follows: and comparing the defect labels belonging to the same category, and if IoU is greater than a set threshold tau and the defect confidence coefficient on the picture to be detected is less than a set threshold gamma, considering the defect labels on the picture to be detected as texture false detection and removing the texture false detection, thereby obtaining the final defect position labels and the corresponding categories of the picture to be detected.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the method adopts deep learning target detection as an integral detection frame, reduces the problem of algorithm quality reduction caused by ambient illumination, camera exposure and displacement, and improves the stability of the algorithm for detecting the surface defects of industrial products.
2. The method provided by the invention cuts data with larger picture size, increases the training data amount, reduces the requirement on video memory in the training process, enables the picture to keep the original size input during training, and is not easy to lose the characteristics of tiny flaws. Meanwhile, the method for detecting the combined defect frame by using the sliding window during detection ensures the uniformity and the integrity of the output of the defect marking frame.
3. In order to improve the difference of the detector on the background patterns and the defects of the texture textiles, in addition to the traditional deep learning data enhancement method, before each iteration of each picture, a mixed splicing mode of the defect pictures and the corresponding normal pictures of the texture patterns is used as an online data enhancement method, so that in the training process, under the condition that the data volume is not increased in a large scale, the diversity of negative samples is enhanced, and the false detection rate of the detector is reduced.
4. The texture template picture is used for pre-detection in the detection process, and the false detection position generated by the texture template is relatively fixed, so that the false detection of small defects generated by the texture template at certain fixed positions on the surface of an industrial product can be eliminated by comparing the defect positions and types of the picture result to be detected and the template result thereof, and the overall identification accuracy is improved.
Drawings
FIG. 1 is a training flow diagram of a sample-based enhanced industrial product surface defect detection method.
FIG. 2 is a defect detection flow chart of a sample enhancement based industrial product surface defect detection method.
Detailed Description
The present invention will be further described with reference to the following specific examples.
The example uses the real collected data of textile pattern with patterns, which includes 15 defects such as stain, stitch mark, hole, etc., the pattern template has 68 kinds, including each template picture, several normal pictures and defect pictures with marks, and the picture size is 4096 x 1810 to 4096 x 1696.
As shown in fig. 1 and fig. 2, the method for detecting surface defects of an industrial product based on sample enhancement provided by the present embodiment includes the following steps:
1) and carrying out size standardization operation on the textile picture set with the patterns, wherein the pictures containing the defects have corresponding defect marking files, cutting the defect pictures and the corresponding defect marks of each picture, and dividing the pictures into a normal picture set and a defect picture set according to the cut marks.
The textile data set with patterns comprises a defect picture set X, a normal picture set Y and a pattern template picture set Z, wherein the defect picture set X contains marks, and each mark of each defect is a rectangular mark frame with a format of (name, category, X)min,ymin,xmax,ymax) Where name represents the picture name, category represents the type of defect, (x)min,ymin) The horizontal and vertical coordinates (x) of the upper left corner of the rectangular labeling boxmax,ymax) And the horizontal and vertical coordinates of the lower right corner of the rectangular labeling frame are represented, and the image sets Y and Z have no labeling information. The three sets of pictures were normalized in size so that all pictures were RGB pictures of 4096 x 1810.
The three picture sets are equally cut, and the defect marks on the picture set X are cut according to the following rules: and cutting the defective rectangular marking frame in the same way as the picture, mapping the defective rectangular marking frame into the range of the cut small picture, if the marking is cut off, calculating the proportion of the area of the cut rectangular marking frame to the area of the original rectangular marking frame, if the area is greater than a set threshold value epsilon, namely 0.25, keeping the marking, and otherwise, discarding the marking.
2) Normalizing and enhancing online random data of the defect picture set obtained in the step 1), wherein the normalization and online random data enhancement comprise random turning up and down, left and right and dividing the defect picture set into batches. The method specifically comprises the following steps: the value 0 is filled in the multiple of the length and the width of 32, the pictures are normalized, a series of operations of random inversion is carried out with the probability of 50%, and then batch processing is carried out, so that the training of the network is facilitated, the training batch number used in the example is 1, namely, one picture is obtained in each batch, and the size of the picture is 1024 x 928.
3) And 2) for each defect picture in the batch in the step 2), randomly searching a normal picture which corresponds to the same normal picture and has the same cutting position and the same texture template pattern in the normal picture set, performing left or right splicing operation on the normal picture and the defect picture, and correspondingly modifying the label file.
3.1) for the defect picture x which is randomly flipped in step 2)i∈XnewFinding out the pattern and cutting position according to the templateCorresponding normal picture set M e (Y)new,Znew) So that x isiAnd the texture template and cut position are the same for all samples in M, where XnewFor a set of cut defect pictures, Ynew,ZnewRespectively a normal picture set and a template picture set after cutting.
3.2) randomly selecting a normal picture yiE is M, for yiIs filled with a value of 0, such that yiIs also filled to 1024 x 928 for yiAnd (6) carrying out normalization processing.
3.3) to yiAnd performing data enhancement operation, namely randomly turning up, down, left and right.
3.4) generating a random number between (0,1) such that x is given a probability of 50%new＝(xi,yi) Another 50% probability is such that xnew＝(yi,xi) I.e. randomly splicing left or right, where xnewThe new samples generated are shown, and the new sample size after splicing is 2048 x 928.
3.5) processing the marking information according to the splicing mode, if the picture x has defectsiOn the left, the defect label does not need to be changed, if the defect picture xiOn the right, 1024 is added to both the minimum and maximum of the broadside coordinates in the rectangular box label.
4) And (3) carrying out iterative training on the pictures and labels of each batch obtained in the step 3) by using a Cascade-RCNN algorithm, and finishing a round of training after all batches are trained.
The Cascade-RCNN algorithm comprises a trunk network, a region proposing network and a classification regression network, which are respectively used for extracting features, generating region propositions and classifying and fine-tuning candidate frames. In the invention, a convolutional neural network ResNeXt-101 and a characteristic pyramid FPN are used as a backbone network, a region proposing network in a two-stage target detector, namely, fast-RCNN, is used as a region proposing network, and a multi-layer cascade network is used as a classification regression network. And training all the enhanced pictures for one round, and finishing one round of training.
5) And after finishing one round of training, repeating the steps 2) to 4) until reaching the set iteration round, wherein the set iteration round is 12 in the example, and outputting and storing the parameters in the network to obtain the Cascade-RCNN detection model.
6) And 5) carrying out sliding window detection on the textile picture to be detected and the pattern template picture determined to be free of defects by using the Cascade-RCNN detection model obtained in the step 5), splicing the results detected by the sliding window, and comparing the results obtained by the sliding window and the pattern template picture to obtain the defect type and the area label of the picture to be detected.
6.1) setting the size of a sliding window to be 1024 x 905 for a picture of the surface of an industrial product to be detected, performing sliding window detection on the picture to be detected by using the Cascade-RCNN detection model obtained in the step 5), and mapping the result back to the region of an original image to obtain the label format (category, x) of each defectmin,ymin,xmax,ymaxScore), category, indicates the defect class, (x)min,ymin) The horizontal and vertical coordinates (x) of the upper left corner of the rectangular labeling boxmax,ymax) And the horizontal and vertical coordinates of the lower right corner of the rectangular labeling frame are represented, the score represents the confidence coefficient of defect judgment, and the value of the confidence coefficient is (0, 1).
6.2) judging the defect labels close to the edges of the sliding windows, wherein if the adjacent sliding windows have labels with the same category and similar position and size, the adjacent standard of the judgment label is as follows: the distance between the rectangular marking frame and the boundary position of the picture is less than 20 pixels, and the shortest distance between the rectangular marking frame and the other marking frame is less than 30 pixels. And then, carrying out annotation merging operation on the holding annotation frames meeting the conditions according to the confidence degree sequence of the holding annotation frames, and merging the defective rectangular annotation frames to allow at most one annotation in each cutting area to participate. The merged rectangle frame is the minimum bounding rectangle of the plurality of rectangle frames, and the new confidence calculation takes the mean value of the rectangle frames as follows:
scorenew＝(score1+score2+...+scoren)/n
wherein, scoreiAnd representing the confidence of the ith rectangular labeling box participating in synthesis, and n represents the total number of the rectangular boxes participating in synthesis.
6.3) adopting the steps 6.1) and 6.2) to carry out defect detection on the pattern template picture Z of the textile with the patterns in advance under the offline condition, and storing the obtained result.
6.4) adopting the steps 6.1) and 6.2) to carry out defect detection on the textile picture with the patterns needing to be detected on line, comparing the obtained result with the result of the corresponding template stored in the step 6.3), and adopting an interaction over Union (IoU) as a comparison standard, wherein the calculation formula is as follows:
Figure BDA0002344783280000091
wherein DR represents a defect rectangular marking frame detected on the to-be-detected image, and GT represents a real defect rectangular marking frame. The specific method for comparison is to compare the defect labels belonging to the same category, and if IoU is greater than a set threshold τ of 0.5 and the confidence of the defect on the picture to be detected is less than a set threshold γ of 0.3, the defect label on the picture to be detected is regarded as a texture false detection and is removed, so as to obtain the final defect position label and the corresponding category of the picture to be detected.
The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be regarded as equivalent substitutions, and are included in the scope of the present invention.

Claims (5)

1. The industrial product surface defect detection method based on sample enhancement is characterized by comprising the following steps:
1) carrying out size standardization operation on a picture set on the surface of an industrial product, wherein the picture containing the defects has a corresponding defect marking file, carrying out cutting operation on the defect picture and the defect marking file corresponding to each picture, and dividing the defect picture and the defect marking file into a normal picture set and a defect picture set according to the cut marks;
2) normalizing and enhancing online random data of the defect picture set obtained in the step 1), including randomly turning the defect picture set up and down, left and right, and dividing the defect picture set into batches;
3) for each defective picture in the batch in the step 2), randomly searching a normal picture which corresponds to the defective picture and has the same cutting position and the same texture template pattern in the normal picture set, performing left or right splicing operation on the normal picture and the defective picture, and correspondingly modifying the label file;
4) performing iterative training on the pictures and labels of each batch obtained in the step 3) by using a Cascade-RCNN algorithm, and finishing a round of training after finishing training all batches;
5) after finishing one round of training, repeating the steps 2) to 4) until reaching the set iteration round, outputting and storing parameters in the network to obtain a Cascade-RCNN detection model;
6) and 5) performing sliding window detection on the surface picture of the industrial product to be detected and the determined non-defective texture template picture by using the Cascade-RCNN detection model obtained in the step 5), splicing the results detected by the sliding window, and comparing the results obtained by the sliding window and the determined non-defective texture template picture to obtain the defect type and the region label of the picture to be detected finally.
2. The method for detecting surface defects of industrial products based on sample enhancement as claimed in claim 1, wherein: in step 1), the image set on the surface of the industrial product comprises a template image set Z consisting of a defect-containing image set X, a defect-free normal image set Y and a pattern example image of each texture template, wherein the defect image set X contains labels, and each defect label is a rectangular label frame with a format of (name, category, X)min,ymin,xmax,ymax) Where name represents the picture name, category represents the type of defect, (x)min,ymin) The horizontal and vertical coordinates (x) of the upper left corner of the rectangular labeling boxmax,ymax) The horizontal and vertical coordinates of the lower right corner of the rectangular labeling frame are represented, and neither the picture set Y nor the picture set Z has labeling information; carrying out size standardization on the three picture sets to enable all pictures to be RGB pictures of H x W, wherein H and W are the height and width of the pictures;
the same average cutting is carried out on the three picture sets, and the defect marks on the picture set X are cut according to the following rules: cutting the defective rectangular marking frame in the same way as the picture, mapping the defective rectangular marking frame into the range of the cut small picture, if the marking is cut off, calculating the proportion of the area of the cut rectangular marking frame to the area of the original rectangular marking frame, if the area is larger than a set threshold epsilon, keeping the marking, and otherwise, discarding;
dividing the picture into a defect picture set X according to the cut marking informationnewSet of normal pictures YnewAnd template picture set ZnewAnd stored according to the position where they were cut.
3. The method for detecting surface defects of industrial products based on sample enhancement as claimed in claim 1, wherein: the step 3) comprises the following steps:
3.1) for the defect picture x which is randomly flipped in step 2)i∈XnewFinding out corresponding normal picture set M e (Y) according to template texture and cutting positionnew,Znew) So that x isiAnd the texture template and cut position are the same for all samples in M, where XnewFor a set of cut defect pictures, Ynew,ZnewRespectively a normal picture set and a template picture set after cutting;
3.2) randomly selecting a normal picture yiE.g. M, according to xiSize of (a) to yiIs filled with a value of 0, such that yiSize and x ofiAre all the same in size and for yiCarrying out normalization processing;
3.3) to yiCarrying out data enhancement operation, namely randomly turning up, down, left and right;
3.4) generating a random number between (0,1) such that x is given a probability of 50%new＝(xi,yi) Another 50% probability is such that xnew＝(yi,xi) I.e. randomly splicing left or right, where xnewRepresenting the generated new sample;
3.5) processing the labeling information according to the splicing modeIf there is a defect in the picture xiOn the left, there is no need to change the defect label, if the defect picture xiAnd on the right, the rectangular marking box needs to be corrected correspondingly.
4. The method for detecting surface defects of industrial products based on sample enhancement as claimed in claim 1, wherein: in step 4), the Cascade-RCNN algorithm comprises a trunk network, a region proposing network and a classification regression network, which are respectively used for extracting features, generating region suggestions and classifying and fine-tuning candidate frames; the convolutional neural network ResNeXt-101 and the feature pyramid FPN are used as a backbone network, the area proposal network uses an area proposal network part in a two-stage target detector, namely, fast-RCNN, and the classification regression network uses a multilayer cascade network.
5. The method for detecting surface defects of industrial products based on sample enhancement as claimed in claim 1, wherein: in step 6), the following detection process is performed:
6.1) for an industrial product surface picture to be detected, using a preset sliding window size, using the Cascade-RCNN detection model obtained in the step 5) to perform sliding window detection on the picture to be detected, and mapping the result back to the area of the original image to obtain the label format (category, x) of each defectmin,ymin,xmax,ymaxScore), category denotes the defect class, (x)min,ymin) The horizontal and vertical coordinates (x) of the upper left corner of the rectangular labeling boxmax,ymax) The horizontal and vertical coordinates of the lower right corner of the rectangular labeling frame are represented, score represents the confidence coefficient of defect judgment, and the value of the confidence coefficient is (0, 1);
6.2) judging the defect labels close to the edges of the sliding windows, if the adjacent sliding windows have labels with the same category and the similar position size, carrying out label merging operation according to the confidence degree sequence, wherein the merged rectangular label frame is the minimum circumscribed rectangle of a plurality of rectangular label frames, and calculating the new confidence degree to obtain the average value of the minimum circumscribed rectangle, which is as follows:
scorenew＝(score1+score2+...+scoren)/n
wherein, scorenewIndicates the new confidence, scoreiRepresenting the confidence of the ith rectangular labeling frame participating in synthesis, and n representing the total number of the rectangular labeling frames participating in synthesis;
6.3) adopting the steps 6.1) and 6.2) to carry out defect detection on the template picture set Z in advance under the offline condition, and storing the obtained result;
6.4) detecting the surface picture of the industrial product to be detected online by adopting the steps 6.1) and 6.2), comparing the obtained result with the detection result of the corresponding template stored in the step 6.3), adopting IoU as a comparison standard, and obtaining the following calculation formula:
Figure FDA0002344783270000041
wherein DR represents a defect rectangular marking frame detected on the to-be-detected image, and GT represents a real defect rectangular marking frame; the specific method for comparison is as follows: and comparing the defect labels belonging to the same category, and if IoU is greater than a set threshold tau and the defect confidence coefficient on the picture to be detected is less than a set threshold gamma, considering the defect labels on the picture to be detected as texture false detection and removing the texture false detection, thereby obtaining the final defect position labels and the corresponding categories of the picture to be detected.
CN201911390407.9A 2019-12-30 2019-12-30 Industrial product surface defect detection method based on sample enhancement Active CN111161243B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911390407.9A CN111161243B (en) 2019-12-30 2019-12-30 Industrial product surface defect detection method based on sample enhancement

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911390407.9A CN111161243B (en) 2019-12-30 2019-12-30 Industrial product surface defect detection method based on sample enhancement

Publications (2)

Publication Number Publication Date
CN111161243A true CN111161243A (en) 2020-05-15
CN111161243B CN111161243B (en) 2023-04-07

Family

ID=70559383

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911390407.9A Active CN111161243B (en) 2019-12-30 2019-12-30 Industrial product surface defect detection method based on sample enhancement

Country Status (1)

Country Link
CN (1) CN111161243B (en)

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111709452A (en) * 2020-05-21 2020-09-25 五邑大学 Evaluation method of wine bottle surface defect model, electronic device and storage medium
CN111860508A (en) * 2020-07-28 2020-10-30 平安科技（深圳）有限公司 Image sample selection method and related equipment
CN111862089A (en) * 2020-08-03 2020-10-30 上海闻泰信息技术有限公司 Method and device for identifying defects of dust screen and electronic equipment
CN111882520A (en) * 2020-06-16 2020-11-03 歌尔股份有限公司 Screen defect detection method and device and head-mounted display equipment
CN112508857A (en) * 2020-11-17 2021-03-16 华南理工大学 Aluminum product surface defect detection method based on improved Cascade R-CNN
CN112700435A (en) * 2021-01-12 2021-04-23 华南理工大学 Wall defect detection method based on deep learning
CN113192061A (en) * 2021-05-25 2021-07-30 安徽睿微信息科技有限公司 LED package appearance detection image extraction method and device, electronic equipment and storage medium
CN113204868A (en) * 2021-04-25 2021-08-03 中车青岛四方机车车辆股份有限公司 Defect detection parameter optimization method and optimization system based on POD quantitative analysis
CN113222990A (en) * 2021-06-11 2021-08-06 青岛高重信息科技有限公司 Chip counting method based on image data enhancement
CN113643235A (en) * 2021-07-07 2021-11-12 青岛高重信息科技有限公司 Chip counting method based on deep learning
CN114120209A (en) * 2022-01-27 2022-03-01 深圳市博铭维技术股份有限公司 Pipeline defect detection method, system, equipment and storage medium
CN114332073A (en) * 2022-01-14 2022-04-12 湖南视比特机器人有限公司 Target workpiece detection method and device, intelligent sorting system and storage medium
CN114612471A (en) * 2022-05-10 2022-06-10 佛山市阿瑞斯数字设备有限公司 Method for detecting surface defects of ceramic tiles with complex textures
CN114998236A (en) * 2022-05-25 2022-09-02 征图新视(江苏)科技股份有限公司 Defect extraction algorithm based on target detection
CN115797314A (en) * 2022-12-16 2023-03-14 哈尔滨耐是智能科技有限公司 Part surface defect detection method, system, equipment and storage medium
CN115965582A (en) * 2022-11-22 2023-04-14 哈尔滨岛田大鹏工业股份有限公司 Ultrahigh-resolution-based engine cylinder body and cylinder cover surface defect detection method
CN115965872A (en) * 2022-07-22 2023-04-14 中科三清科技有限公司 Tea leaf picking method and device, electronic equipment and storage medium
WO2024044915A1 (en) * 2022-08-29 2024-03-07 西门子股份公司 Image comparison method and apparatus for error detection, and computer device

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108663431A (en) * 2014-05-18 2018-10-16 查尔斯斯塔克德雷珀实验室有限公司 The method for detecting the defects of ferromagnetic material
CN109829893A (en) * 2019-01-03 2019-05-31 武汉精测电子集团股份有限公司 A kind of defect object detection method based on attention mechanism

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108663431A (en) * 2014-05-18 2018-10-16 查尔斯斯塔克德雷珀实验室有限公司 The method for detecting the defects of ferromagnetic material
CN109829893A (en) * 2019-01-03 2019-05-31 武汉精测电子集团股份有限公司 A kind of defect object detection method based on attention mechanism

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
徐秀 等: "基于Faster R-CNN的圆柱形金属工件表面缺陷检测", 《软件导刊》 *

Cited By (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111709452A (en) * 2020-05-21 2020-09-25 五邑大学 Evaluation method of wine bottle surface defect model, electronic device and storage medium
CN111709452B (en) * 2020-05-21 2023-06-16 五邑大学 Method for evaluating surface defect model of wine bottle, electronic device and storage medium
CN111882520A (en) * 2020-06-16 2020-11-03 歌尔股份有限公司 Screen defect detection method and device and head-mounted display equipment
CN111882520B (en) * 2020-06-16 2023-10-17 歌尔光学科技有限公司 Screen defect detection method and device and head-mounted display equipment
CN111860508A (en) * 2020-07-28 2020-10-30 平安科技（深圳）有限公司 Image sample selection method and related equipment
CN111862089A (en) * 2020-08-03 2020-10-30 上海闻泰信息技术有限公司 Method and device for identifying defects of dust screen and electronic equipment
CN112508857B (en) * 2020-11-17 2022-04-22 华南理工大学 Aluminum product surface defect detection method based on improved Cascade R-CNN
CN112508857A (en) * 2020-11-17 2021-03-16 华南理工大学 Aluminum product surface defect detection method based on improved Cascade R-CNN
CN112700435A (en) * 2021-01-12 2021-04-23 华南理工大学 Wall defect detection method based on deep learning
CN113204868A (en) * 2021-04-25 2021-08-03 中车青岛四方机车车辆股份有限公司 Defect detection parameter optimization method and optimization system based on POD quantitative analysis
CN113204868B (en) * 2021-04-25 2023-02-28 中车青岛四方机车车辆股份有限公司 Defect detection parameter optimization method and optimization system based on POD quantitative analysis
CN113192061A (en) * 2021-05-25 2021-07-30 安徽睿微信息科技有限公司 LED package appearance detection image extraction method and device, electronic equipment and storage medium
CN113192061B (en) * 2021-05-25 2023-08-29 安徽睿微信息科技有限公司 Extraction method and device of LED package appearance detection image, electronic equipment and storage medium
CN113222990A (en) * 2021-06-11 2021-08-06 青岛高重信息科技有限公司 Chip counting method based on image data enhancement
CN113222990B (en) * 2021-06-11 2023-03-14 青岛高重信息科技有限公司 Chip counting method based on image data enhancement
CN113643235A (en) * 2021-07-07 2021-11-12 青岛高重信息科技有限公司 Chip counting method based on deep learning
CN113643235B (en) * 2021-07-07 2023-12-29 青岛高重信息科技有限公司 Chip counting method based on deep learning
CN114332073A (en) * 2022-01-14 2022-04-12 湖南视比特机器人有限公司 Target workpiece detection method and device, intelligent sorting system and storage medium
CN114120209B (en) * 2022-01-27 2022-05-20 深圳市博铭维技术股份有限公司 Pipeline defect detection method, system, equipment and storage medium
CN114120209A (en) * 2022-01-27 2022-03-01 深圳市博铭维技术股份有限公司 Pipeline defect detection method, system, equipment and storage medium
CN114612471A (en) * 2022-05-10 2022-06-10 佛山市阿瑞斯数字设备有限公司 Method for detecting surface defects of ceramic tiles with complex textures
CN114998236A (en) * 2022-05-25 2022-09-02 征图新视(江苏)科技股份有限公司 Defect extraction algorithm based on target detection
CN115965872A (en) * 2022-07-22 2023-04-14 中科三清科技有限公司 Tea leaf picking method and device, electronic equipment and storage medium
CN115965872B (en) * 2022-07-22 2023-08-15 中科三清科技有限公司 Tea picking method and device, electronic equipment and storage medium
WO2024044915A1 (en) * 2022-08-29 2024-03-07 西门子股份公司 Image comparison method and apparatus for error detection, and computer device
CN115965582A (en) * 2022-11-22 2023-04-14 哈尔滨岛田大鹏工业股份有限公司 Ultrahigh-resolution-based engine cylinder body and cylinder cover surface defect detection method
CN115965582B (en) * 2022-11-22 2024-03-08 哈尔滨岛田大鹏工业股份有限公司 Ultrahigh-resolution-based method for detecting surface defects of cylinder body and cylinder cover of engine
CN115797314A (en) * 2022-12-16 2023-03-14 哈尔滨耐是智能科技有限公司 Part surface defect detection method, system, equipment and storage medium
CN115797314B (en) * 2022-12-16 2024-04-12 哈尔滨耐是智能科技有限公司 Method, system, equipment and storage medium for detecting surface defects of parts

Also Published As

Publication number Publication date
CN111161243B (en) 2023-04-07

Similar Documents

Publication Publication Date Title
CN111161243B (en) Industrial product surface defect detection method based on sample enhancement
CN109961049B (en) Cigarette brand identification method under complex scene
CN115082683B (en) Injection molding defect detection method based on image processing
CN115082419B (en) Blow-molded luggage production defect detection method
CN110276754B (en) Surface defect detection method, terminal device and storage medium
CN108562589B (en) Method for detecting surface defects of magnetic circuit material
CN113160192B (en) Visual sense-based snow pressing vehicle appearance defect detection method and device under complex background
CN114140679B (en) Defect fusion method, device, recognition system and storage medium
CN111915704A (en) Apple hierarchical identification method based on deep learning
CN109308700A (en) A kind of visual identity defect inspection method based on printed matter character
CN112508857B (en) Aluminum product surface defect detection method based on improved Cascade R-CNN
CN108181316B (en) Bamboo strip defect detection method based on machine vision
CN114972356B (en) Plastic product surface defect detection and identification method and system
CN116559183B (en) Method and system for improving defect judging efficiency
CN113706490B (en) Wafer defect detection method
CN111461133B (en) Express delivery surface single item name identification method, device, equipment and storage medium
CN114119591A (en) Display screen picture quality detection method
CN113034464A (en) Method for detecting defects of liquid crystal display in real time under multiple backgrounds
CN114359235A (en) Wood surface defect detection method based on improved YOLOv5l network
CN115078365A (en) Soft package printing quality defect detection method
CN114627089A (en) Defect identification method, defect identification device, computer equipment and computer readable storage medium
CN114255212A (en) FPC surface defect detection method and system based on CNN
CN111754502A (en) Method for detecting surface defects of magnetic core based on fast-RCNN algorithm of multi-scale feature fusion
CN116664540A (en) Rubber sealing ring surface defect detection method based on Gaussian line detection
CN115861259A (en) Lead frame surface defect detection method and device based on template matching

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant