CN111627018A - Steel plate surface defect classification method based on double-flow neural network model - Google Patents

Steel plate surface defect classification method based on double-flow neural network model Download PDF

Info

Publication number
CN111627018A
CN111627018A CN202010484073.8A CN202010484073A CN111627018A CN 111627018 A CN111627018 A CN 111627018A CN 202010484073 A CN202010484073 A CN 202010484073A CN 111627018 A CN111627018 A CN 111627018A
Authority
CN
China
Prior art keywords
defect
steel plate
neural network
detected
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010484073.8A
Other languages
Chinese (zh)
Other versions
CN111627018B (en
Inventor
倪红军
张加俏
任福继
张振亚
贯大兴
吕帅帅
汪兴兴
张福豹
朱昱
张守阳
王凯旋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nantong University
Original Assignee
Nantong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nantong University filed Critical Nantong University
Priority to CN202010484073.8A priority Critical patent/CN111627018B/en
Publication of CN111627018A publication Critical patent/CN111627018A/en
Application granted granted Critical
Publication of CN111627018B publication Critical patent/CN111627018B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G01N2021/8854—Grading and classifying of flaws
    • G01N2021/8861—Determining coordinates of flaws
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G01N2021/8854—Grading and classifying of flaws
    • G01N2021/8874—Taking dimensions of defect into account
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G01N2021/8854—Grading and classifying of flaws
    • G01N2021/888—Marking defects
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G01N2021/8887—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges based on image processing techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20048—Transform domain processing
    • G06T2207/20061—Hough transform
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30136—Metal
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Pathology (AREA)
  • Biochemistry (AREA)
  • Analytical Chemistry (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Chemical & Material Sciences (AREA)
  • Signal Processing (AREA)
  • Immunology (AREA)
  • Computing Systems (AREA)
  • Quality & Reliability (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)
  • Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)

Abstract

The invention provides a steel plate surface defect classification method based on a double-flow neural network model, which comprises the following steps of: s10, establishing a standard map library of the surface defects of the steel plate; s20 classifying the defects of the pictures to be detected, importing the pictures to be detected into a double-flow neural network to obtain a global priority value and a local characteristic value, and determining the defect category of each picture to be detected by integrating the global priority value and the local characteristic value; and S30 judging the defect position of the picture to be detected, and determining the position of the defect category in the classification result by using a YOLO network, wherein two residual error network modules in the YOLO network are replaced by two feature multiplexing network modules. The invention relates to a steel plate surface defect classification method of a double-flow neural network model, which comprises the steps of firstly judging the type of a defect by adopting the double-flow neural network model, and then importing a picture to be detected into a YOLO network to predict the position of the defect; the double-current neural network and the YOLO network are matched to improve the defect positioning precision and improve the detection efficiency.

Description

Steel plate surface defect classification method based on double-flow neural network model
Technical Field
The invention relates to the technical field of steel plate defect detection, in particular to a steel plate surface defect classification method based on a double-current neural network model.
Background
Sheet metal is an essential raw material in the mechanical industry, and the product quality of the sheet metal is a key index for determining the price of the sheet metal. Due to the problems of limited equipment and process conditions and the like, different types and categories of defects inevitably exist on the surface of the metal plate, and the size, the number and the distribution of the defects are greatly different. Due to the diversity and complexity of surface defects, steel manufacturers pay great attention to the detection of surface quality, and it is not only expensive to improve the detection technology and improve the detection level.
The common defects on the surface of the steel plate can be divided into two categories according to the shape of the defects, namely planar defects and linear defects. Common deep learning models such as a retnet model, a spatial pyramid model and the like are all used for directly reading images, extracting image features through convolution operation and then predicting defect types. This algorithm only takes into account the local features of the defect during the convolution process.
The YOLO network model was proposed as early as 2016, and its later version YOLOv3 is not only faster in detection speed, but also more suitable for detection of small targets. The YOLO network contains 24 volume base layers, 4 max pooling layers and two fully connected layers. The volume base layer is used to obtain image features, the max pooling layer is used to reduce image pixels, and the full link layer is used to predict image categories and locations. The YOLO uses the features of the full map to predict the bounding box and classify the target in the box, which means that the YOLO network can use the full map information to realize the target classification and the target position detection in the same image.
In the process of image detection, the YOLO can realize classification and detection of targets through multilayer convolution. For example, when a dog and a cat exist in one picture, the YOLO network can classify the dog and the cat to distinguish which is the dog and which is the cat, and can position the position and mark the target position by using a square frame. The target detection result is evaluated using the confidence value, and the calculation formula is shown below. It can be seen that the confidence value is the product of the classification probability Pr and the IOU value, both belonging to [0,1 ]. The IOU value is the intersection ratio of the area of the prediction box and the area of the real box.
The invention patent of publication No. CN110490842A discloses a strip steel surface defect detection method based on deep learning, which extracts local information of the strip steel surface through a defect judgment and defect classification double-flow network model, and performs comprehensive analysis by combining a scale pyramid to obtain a class chart, and finally obtains the type and position of a defect at the same time, wherein the defect judgment and defect classification double-flow network model comprises a defect judgment branch and a defect classification branch.
Although the method can detect defects, the method can only detect one type of defect, only feed back one type of defect on the surface of a steel plate with more than two types of defects, and provide insufficient data for guiding production. In actual production, the problem in actual production can not be solved easily because of inaccurate judgment, so that the defects exist for a long time and must be fed back through manual detection. Causing waste of manpower and material resources.
Disclosure of Invention
In order to solve the problems, the invention provides a steel plate surface defect classification method based on a double-current neural network model, which comprises the steps of firstly judging the type of a defect by adopting the double-current neural network model, and then leading a picture to be detected into a YOLO network to predict the position of the defect; the double-current neural network and the YOLO network are matched to improve the defect positioning precision and improve the detection efficiency.
In order to achieve the above purpose, the invention adopts a technical scheme that:
a steel plate surface defect classification method based on a double-flow neural network model comprises the following steps: s10, establishing a standard map library of the surface defects of the steel plate; s20 classifying the defects of the pictures to be detected, importing the pictures to be detected into a double-flow neural network to obtain a global priority value and a local characteristic value, and determining the defect category of each picture to be detected by integrating the global priority value and the local characteristic value; and S30 judging the defect position of the picture to be detected, and determining the position of the defect category in the classification result by using a YOLO network, wherein two residual error network modules in the YOLO network are replaced by two feature multiplexing network modules.
Further, the step S10 includes the following steps: s11, each picture in the defect standard image library contains a typical defect, and the pictures are corrected and cut by using Hough transform so that the size of the image is 200 × 200 dpi; and S12, labeling the image by using labelImg software, labeling the position of the defect in the image by using a rectangular real frame, recording coordinate information of the upper left corner (xL, yL) and the lower right corner (xR, yR) of the rectangular frame, and labeling each individual defect by adopting a dense labeling method in the labeling process.
Further, the step S20 includes the following steps: s21, importing the picture to be detected into a double-flow neural network, extracting image features by the double-flow neural network, and distributing the extracted image features according to a certain weight to obtain two flow directions; s22, the weight of one image feature enters a global priority network, a global feature prior is captured from the whole image, the global defect category is predicted in the large direction, and finally a global priority value y1 is obtained through the global priority network; s23, the other image feature weight enters a space pyramid convolution layer, firstly, the space pyramid convolution layer extracts multi-scale example image features, then, each generated example feature is mapped through a full connection layer, and finally, the related region in the mapping is selected through a space pooling layer to carry out corresponding local defect type prediction of the global defect type, so that a local feature value y2 is obtained; and S24, aggregating the global priority value and the local characteristic value of the picture to be detected by using a polymer layer, and synthesizing the global priority value and the local characteristic value to obtain a defect type prediction result.
Further, the global priority network is based on a VGG-16network architecture and comprises a 2 × 2 pooling layer and 3 full connection layers FCa, FCb and FCg, and a bypass connection is arranged between the FCa and the FCg, so that FCa bypasses the FCb and is directly connected with the FCg.
Further, the global defect category comprises a planar defect and a linear defect; the local defect type of the planar defect comprises: at least one of a patch (Pa), a surface Pitting (PS), and a scale indentation (RS); the local defect classes of the linear defect include: at least one of reticulation (Cr), inclusions (In), and scratches (Sc).
Further, two residual error network modules in the YOLO network are replaced by two feature multiplexing network modules for use, each feature multiplexing network module includes 3 convolutional layers, each convolutional layer can obtain the output of all previous convolutional layers as input, and adjacent convolutional layers are connected through the convolutional layers and the pooling layers.
Further, the step S30 includes the following steps: s31, importing the steel plate images to be detected into a YOLO network based on classification priority, and unifying the size of the steel plate images to be detected into 448 x 448dpi by adopting a bilinear interpolation method; s32, carrying out normalization processing on the steel plate image to be detected after the size is adjusted, converting the value range of the pixel value of the steel plate image to be detected from [0,255] into [0,1], and obtaining the first defect classification map; the normalized formula is:
Figure BDA0002518303980000031
wherein xi represents the image pixel point value, min (x), max (x) represent the maximum and minimum values of the image pixel; and S33 dividing the first defect classification map into S × S grids, if the center of the target defect falls into a grid cell, the grid cell is responsible for detecting the object, and obtaining the position detection result of the target defect.
Further, a YOLO network parameter is set, the number K of the cluster clusters is 6, the convolution kernel size is 1 × 1, the convolution step is 1, the initial learning rate of the model is 0.01, the number of samples selected in one training is 4, the weight attenuation regularization term is set to 0.0005, and asynchronous random gradient descent with a momentum term of 0.9 is adopted.
Further, each grid predicts B prediction boxes; the prediction box contains 5 data values (x, y, w, h, confidence); (x, y) is the offset of the center of the prediction box with respect to the current grid, (w, h) is the length and width of the prediction box; the confidence value reflects whether the bounding box contains the target probability and the situation that the current bounding box is coincident with the real bounding box
Figure BDA0002518303980000041
The final detection result conforms to the following formula:
Figure BDA0002518303980000042
wherein pr (object) is an object determination parameter, and pr (object) is 1 when the classification result has an object defect, and pr (object) is 0 when the classification result has no object defect; pr (classic/object) is a conditional probability of a certain class.
Compared with the prior art, the technical scheme of the invention has the following advantages:
the invention relates to a steel plate surface defect classification method based on a double-flow neural network model, wherein one flow of the double-flow neural network is used for judging the global defect type, and the other flow of the double-flow neural network is used for judging the local defect under a certain global defect type; firstly, judging the type of the defect by adopting a double-current neural network model, and then importing the picture to be detected into a YOLO network to predict the position of the defect; the double-current neural network and the YOLO network are matched to improve the defect positioning precision and improve the detection efficiency.
Drawings
The technical solution and the advantages of the present invention will be apparent from the following detailed description of the embodiments of the present invention with reference to the accompanying drawings.
FIG. 1 is a flowchart illustrating a steel plate surface defect classification method based on a dual-flow neural network model according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a steel plate surface defect classification method based on a double-flow neural network model according to an embodiment of the present invention;
FIG. 3 is a diagram of a global priority network architecture according to an embodiment of the present invention;
FIG. 4 illustrates the types of surface defects of a steel plate according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating a method for determining a defect position on a steel plate based on a YOLO network according to an embodiment of the present invention;
FIG. 6 shows the test results of steel plate defect priority classification by a dual-flow neural network according to an embodiment of the present invention;
FIG. 7 shows the results of classification tests performed on the surface defect data sets of steel plates by different algorithms.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment provides a steel plate surface defect classification method based on a double-flow neural network model, as shown in fig. 1-2, the method comprises the following steps: s10, establishing a standard map library of the surface defects of the steel plate. S20 classifying the defects of the pictures to be detected, importing the pictures to be detected into a double-flow neural network to obtain a global priority value and a local characteristic value, and determining the defect type of each picture to be detected by integrating the global priority value and the local characteristic value. And S30 judging the defect position of the picture to be detected, and determining the position of the defect category in the classification result by using a YOLO network, wherein two residual error network modules in the YOLO network are replaced by two feature multiplexing network modules.
The step S10 includes the following steps: s11, each picture in the defect standard library contains a typical defect, and the picture is corrected and cropped by using hough transform to make the image size 200 × 200 dpi. And S12 labeling the image using labelImg software, labeling the position of the defect in the image using a rectangular real frame, and recording the upper left corner (x) of the rectangular frameL,yL) And the lower right corner (x)R,yR) The coordinate information of (2) is marked by adopting a dense marking method in the marking process, namely, each individual defect is marked. The defect standard library comprises the defect types such as texture (Cr), inclusion (in), patches (Pa), surface pock Patterned Surface (PS), scratch patterns (Sc), and the like. Further, defects can be classified into two major categories, namely, planar defects and linear defects, and the local defect categories of the planar defects include patches (Pa), surface Pits (PS), inclusions (In), and the like; the local defect types of the linear defect include a textured pattern (Cr), a scratch pattern (Sc), and the like.
The step S20 includes the following steps: s21, importing the picture to be detected into a double-flow neural network, extracting image features by the double-flow neural network, and distributing the extracted image features according to a certain weight to obtain two flow directions.
S22 the weight of one image feature enters a global priority network, the global feature prior is captured from the whole image, the global defect category is predicted in the large direction, and finally the global priority value y is obtained through the global priority network1As shown in fig. 3, the global priority network is based on VGG-16network architecture, and includes a pooling layer of 2 × 2 and 3 full connection layers FCa, FCb and FCg, where the FC isa is connected to the FCg by a bypass connection, so that FCa is connected directly to the FCg by bypassing the FCb. The FCa and the FCb are fully utilized, the possible information loss is reduced, and a more accurate global priority value is generated. The global priority value y1 is finally obtained through the global priority network. As shown in fig. 4, the global defect categories include a planar defect and a linear defect. Since the linear defect and the planar defect are macroscopically very different, accurate discrimination can be performed by the global priority network.
S23 the other weight of the image feature enters into the space pyramid convolution layer, firstly the space pyramid convolution layer extracts the multi-scale example image feature, then each generated example feature is mapped through a full connection layer, and finally the local defect category prediction of the global defect category is carried out through the relevant area in the space pooling layer selection mapping to obtain the local feature value y2. The local defect type of the planar defect comprises: at least one of plaque (Pa), surface Pock (PS), and inclusions (In); the local defect classes of the linear defect include: at least one of a texture (Cr) and a scratch (Sc).
And S24, aggregating the global priority value and the local characteristic value of the picture to be detected by using a polymer layer, and synthesizing the global priority value and the local characteristic value to obtain a defect type prediction result. The formula is as follows:
Figure BDA0002518303980000061
where y is the aggregate score, W is a weight matrix of c × 2c, b is the variance, and c is the number of classifications. When the global priority network confirms that the defect type is the planar defect, point defect conclusions such as reticulate patterns, scratches and the like cannot be predicted through local prediction, and classification accuracy is improved. Taking the surface pockmarks of the defect as an example, extracting image characteristic information and carrying out weight distribution. If the probability that the defect is the planar defect is predicted to be 0.8 by the global priority network, the probability of the linear defect is 0.2; the local feature network predicts the probability of the defect being a surface pock 0.5 and the probability of the reticulate pattern 0.5. Combining the networks of the two flows, one can eventually get: the probability of the defect being a surface pock is 0.4, and the probability of the defect being a cross hatch is 0.1. The defect can be determined to be a surface pitting.
As shown in fig. 5, the step S30 includes the following steps: s31, importing the steel plate images to be detected into a sorting priority-based YOLO network, and unifying the size of the steel plate images to be detected into 448 x 448dpi by adopting a bilinear interpolation method. The specific method of bilinear interpolation is as follows: if we want to find the value of the unknown function f at point P (x, y), we assume that we know the value of the function f at four points Q11 (x1, y1), Q12 (x1, y2), Q21 (x2, y1) and Q22 (x2, y 2). The final result of bilinear interpolation is:
Figure BDA0002518303980000071
s32, carrying out normalization processing on the steel plate image to be detected after the size is adjusted, converting the value range of the pixel value of the steel plate image to be detected from [0,255] into [0,1], and obtaining the first defect classification map; the normalized formula is:
Figure BDA0002518303980000072
where xi represents the image pixel point value, and min (x), max (x) represent the maximum and minimum values of the image pixel. The information storage of the normalized image is not changed, but the value range of the pixel value of the image is converted from 0,255 to 0,1, so that the subsequent neural network processing is facilitated.
And S33 dividing the first defect classification map into S × S grids, if the center of the target defect falls into a grid cell, the grid cell is responsible for detecting the object, and obtaining the position detection result of the target defect.
Two residual error network modules in the YOLO network are replaced by two feature multiplexing network modules for use, so that the model can receive multilayer convolution features output by dense connection blocks before prediction is carried out. Each feature multiplexing network module comprises 3 convolutional layers, each convolutional layer can obtain the output of all the previous convolutional layers as input, and adjacent convolutional layers are connected through the convolutional layers and the pooling layers. Setting YOLO network parameters, taking the number K of clustering clusters to be 6, the convolution kernel size to be 1 x1, the convolution step length to be 1, the initial learning rate of the model to be 0.01, the number of samples selected in one training to be 4, the weight attenuation regular term to be 0.0005, and adopting asynchronous random gradient descent with the momentum term to be 0.9.
Predicting B prediction boxes by each grid; the prediction box contains 5 data values (x, y, w, h, confidence). (x, y) is the offset of the center of the prediction box with respect to the current mesh, and (w, h) is the length and width of the prediction box. The confidence value reflects whether the bounding box contains the target probability and the situation that the current bounding box is coincident with the real bounding box
Figure BDA0002518303980000073
I.e. comprising two parts: the target object Pr (object) is contained in the grid, and the accuracy of the grid prediction B is improved. Wherein pr (object) is an object determination parameter, and pr (object) is 1 when the classification result has an object defect, and pr (object) is 0 when the classification result has no object defect. Pr (classic/object) is a conditional probability of a certain class.
Wherein IoU is the intersection ratio of the predicted frame and the real frame, and the calculation formula is:
Figure BDA0002518303980000081
when there are C types of defects in the image, the conditional probability of the C types is Pr (classic/object), which indicates the probability that the mesh contains the target object and belongs to the i-th class object. The final output probability is thus
Figure BDA0002518303980000082
The value of Pr (classic) belongs to [0,1], so the defect detection precision of the improved model is higher.
The final detection result conforms to the following formula:
Figure BDA0002518303980000083
the steel plate defects are classified preferentially through the double-flow neural network, and the experimental result is shown in fig. 6. After the network initialization is finished, the classification capability is not available, the accuracy of the training set is 0.65 in the initial stage, and then the accuracy gradually rises along with the increase of the iteration times; the accuracy rate in the test set is gradually increased along with the increase of the iteration times and finally converges to 1; the loss value of the training set and the test set decreases as the training progresses, and finally converges to 0.
As can be seen from fig. 7, the method provided by the present application has the highest accuracy in classifying the defects of the steel plate, and the average test accuracy can reach 99.7%.
The above description is only an exemplary embodiment of the present invention, and not intended to limit the scope of the present invention, and all equivalent structures or equivalent processes that are transformed by the content of the present specification and the attached drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims (9)

1. A steel plate surface defect detection method based on a double-flow neural network is characterized by comprising the following steps:
s10, establishing a standard map library of the surface defects of the steel plate;
s20 classifying the defects of the pictures to be detected, importing the pictures to be detected into a double-flow neural network to obtain a global priority value and a local characteristic value, and determining the defect category of each picture to be detected by integrating the global priority value and the local characteristic value; and
s30 judging the defect position of the picture to be detected, and determining the position of the defect category in the classification result by using a YOLO network, wherein two residual error network modules in the YOLO network are replaced by two feature multiplexing network modules.
2. The method for classifying the surface defects of the steel plate based on the dual-flow neural network model according to claim 1, wherein the step S10 comprises the following steps:
s11, each picture in the defect standard image library contains a typical defect, and the pictures are corrected and cut by using Hough transform so that the size of the image is 200 × 200 dpi; and
s12 labeling the image with labelImg software, labeling the position of the defect in the image with a rectangular real frame, and recording the upper left corner (x) of the rectangular frameL,yL) And the lower right corner (x)R,yR) The coordinate information of (2) is marked by adopting a dense marking method in the marking process, namely, each individual defect is marked.
3. The method for classifying the surface defects of the steel plate based on the dual-flow neural network model according to claim 1, wherein the step S20 comprises the following steps:
s21, importing the picture to be detected into a double-flow neural network, extracting image features by the double-flow neural network, and distributing the extracted image features according to a certain weight to obtain two flow directions;
s22 the weight of one image feature enters a global priority network, the global feature prior is captured from the whole image, the global defect category is predicted in the large direction, and finally the global priority value y is obtained through the global priority network1；
S23 the other weight of the image feature enters into the space pyramid convolution layer, firstly the space pyramid convolution layer extracts the multi-scale example image feature, then each generated example feature is mapped through a full connection layer, and finally the local defect category prediction of the global defect category is carried out through the relevant area in the space pooling layer selection mapping to obtain the local feature value y2(ii) a And
and S24, aggregating the global priority value and the local characteristic value of the picture to be detected by using a polymer layer, and synthesizing the global priority value and the local characteristic value to obtain a defect type prediction result.
4. The dual-flow neural network model-based steel plate surface defect classification method of claim 3, wherein the global priority network is based on VGG-16network architecture and comprises a 2 x2 pooling layer and 3 full-connection layers FCa, FCb and FCg, and a bypass connection is arranged between the FCa and the FCg, so that FCa bypasses FCb and is directly connected with FCg.
5. The method for classifying the surface defects of the steel plate based on the double-current neural network model according to claim 3, wherein the global defect categories comprise planar defects and linear defects; the local defect type of the planar defect comprises: at least one of plaque (Pa), surface Pock (PS), and inclusions (In); the local defect classes of the linear defect include: at least one of a texture (Cr) and a scratch (Sc).
6. The method for classifying the surface defects of the steel plate based on the dual-flow neural network model of claim 1, wherein two residual error network modules in the YOLO network are replaced by two feature multiplexing network modules, each feature multiplexing network module comprises 3 convolutional layers, each convolutional layer can obtain the output of all previous convolutional layers as input, and adjacent convolutional layers are connected through the convolutional layers and the pooling layers.
7. The method for classifying the surface defects of the steel plate based on the dual-flow neural network model according to claim 6, wherein the step S30 comprises the following steps:
s31, importing the steel plate images to be detected into a YOLO network based on classification priority, and unifying the size of the steel plate images to be detected into 448 x 448dpi by adopting a bilinear interpolation method;
s32, carrying out normalization processing on the steel plate image to be detected after the size is adjusted, converting the value range of the pixel value of the steel plate image to be detected from [0,255] into [0,1], and obtaining the first defect classification map; the normalized formula is:
Figure FDA0002518303970000021
wherein xi represents the image pixel point value, min (x), max (x) represent the maximum and minimum values of the image pixel; and
s33, dividing the first defect classification map into S multiplied by S grids, if the center of the target defect falls into a grid unit, the grid unit is responsible for detecting the object, and obtaining the position detection result of the target defect.
8. The method for classifying the surface defects of the steel plate based on the dual-flow neural network model, according to claim 7, setting YOLO network parameters, taking the number K of the cluster as 6, the convolution kernel size as 1 x1, the convolution step as 1, the initial learning rate of the model as 0.01, the number of samples selected in one training as 4, the weight attenuation regular term as 0.0005, and adopting asynchronous random gradient descent with the momentum term as 0.9.
9. The method for classifying the surface defects of the steel plate based on the double-current neural network model according to claim 8, wherein each grid predicts B prediction boxes; the prediction box contains 5 data values (x, y, w, h, confidence); (x, y) is the offset of the center of the prediction box with respect to the current grid, (w, h) is the length and width of the prediction box; the confidence value reflects whether the bounding box contains the target probability and the situation that the current bounding box is coincident with the real bounding box
Figure FDA0002518303970000031
The final detection result conforms to the following formula:
Figure FDA0002518303970000032
wherein pr (object) is an object determination parameter, and pr (object) is 1 when the classification result has an object defect, and pr (object) is 0 when the classification result has no object defect; pr (classic/object) is a conditional probability of a certain class.
CN202010484073.8A 2020-06-01 2020-06-01 Steel plate surface defect classification method based on double-flow neural network model Active CN111627018B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010484073.8A CN111627018B (en) 2020-06-01 2020-06-01 Steel plate surface defect classification method based on double-flow neural network model

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010484073.8A CN111627018B (en) 2020-06-01 2020-06-01 Steel plate surface defect classification method based on double-flow neural network model

Publications (2)

Publication Number Publication Date
CN111627018A true CN111627018A (en) 2020-09-04
CN111627018B CN111627018B (en) 2023-08-04

Family

ID=72272547

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010484073.8A Active CN111627018B (en) 2020-06-01 2020-06-01 Steel plate surface defect classification method based on double-flow neural network model

Country Status (1)

Country Link
CN (1) CN111627018B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114841915A (en) * 2022-03-14 2022-08-02 阿里巴巴（中国）有限公司 Tile flaw detection method and system based on artificial intelligence and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108345911A (en) * 2018-04-16 2018-07-31 东北大学 Surface Defects in Steel Plate detection method based on convolutional neural networks multi-stage characteristics
US20180268257A1 (en) * 2017-03-20 2018-09-20 Rolls-Royce Plc Surface defect detection
CN108596026A (en) * 2018-03-16 2018-09-28 中国科学院自动化研究所 Across the visual angle Gait Recognition device and training method of confrontation network are generated based on double fluid
CN109064461A (en) * 2018-08-06 2018-12-21 长沙理工大学 A kind of detection method of surface flaw of steel rail based on deep learning network
CN110490842A (en) * 2019-07-22 2019-11-22 同济大学 A kind of steel strip surface defect detection method based on deep learning
CN111062915A (en) * 2019-12-03 2020-04-24 浙江工业大学 Real-time steel pipe defect detection method based on improved YOLOv3 model

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180268257A1 (en) * 2017-03-20 2018-09-20 Rolls-Royce Plc Surface defect detection
CN108596026A (en) * 2018-03-16 2018-09-28 中国科学院自动化研究所 Across the visual angle Gait Recognition device and training method of confrontation network are generated based on double fluid
CN108345911A (en) * 2018-04-16 2018-07-31 东北大学 Surface Defects in Steel Plate detection method based on convolutional neural networks multi-stage characteristics
CN109064461A (en) * 2018-08-06 2018-12-21 长沙理工大学 A kind of detection method of surface flaw of steel rail based on deep learning network
CN110490842A (en) * 2019-07-22 2019-11-22 同济大学 A kind of steel strip surface defect detection method based on deep learning
CN111062915A (en) * 2019-12-03 2020-04-24 浙江工业大学 Real-time steel pipe defect detection method based on improved YOLOv3 model

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114841915A (en) * 2022-03-14 2022-08-02 阿里巴巴（中国）有限公司 Tile flaw detection method and system based on artificial intelligence and storage medium

Also Published As

Publication number Publication date
CN111627018B (en) 2023-08-04

Similar Documents

Publication Publication Date Title
CN111612784B (en) Steel plate surface defect detection method based on classification priority YOLO network
CN110660052B (en) Hot-rolled strip steel surface defect detection method based on deep learning
CN111223088B (en) Casting surface defect identification method based on deep convolutional neural network
CN110070008B (en) Bridge disease identification method adopting unmanned aerial vehicle image
CN108918536B (en) Tire mold surface character defect detection method, device, equipment and storage medium
CN111079640B (en) Vehicle type identification method and system based on automatic amplification sample
CN108711148B (en) Tire defect intelligent detection method based on deep learning
CN103221807B (en) Fast processing and the uneven factor detected in web shaped material
CN109544522A (en) A kind of Surface Defects in Steel Plate detection method and system
CN111798447B (en) Deep learning plasticized material defect detection method based on fast RCNN
CN111369526B (en) Multi-type old bridge crack identification method based on semi-supervised deep learning
CN114399719B (en) Transformer substation fire video monitoring method
CN115439458A (en) Industrial image defect target detection algorithm based on depth map attention
CN115272204A (en) Bearing surface scratch detection method based on machine vision
CN115147380A (en) Small transparent plastic product defect detection method based on YOLOv5
CN116109616A (en) Pavement crack detection and small-surface element fitting detection method based on YOLOv5
Hacıefendioğlu et al. Wood construction damage detection and localization using deep convolutional neural network with transfer learning
CN116596875A (en) Wafer defect detection method and device, electronic equipment and storage medium
CN111627018B (en) Steel plate surface defect classification method based on double-flow neural network model
CN113408630A (en) Transformer substation indicator lamp state identification method
CN106682604B (en) Blurred image detection method based on deep learning
CN117392097A (en) Additive manufacturing process defect detection method and system based on improved YOLOv8 algorithm
CN110765900B (en) Automatic detection illegal building method and system based on DSSD
CN116645351A (en) Online defect detection method and system for complex scene
CN115830302A (en) Multi-scale feature extraction and fusion power distribution network equipment positioning identification method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant