CN114299066B - Defect detection method and device based on salient feature pre-extraction and image segmentation - Google Patents
Defect detection method and device based on salient feature pre-extraction and image segmentation Download PDFInfo
- Publication number
- CN114299066B CN114299066B CN202210200679.3A CN202210200679A CN114299066B CN 114299066 B CN114299066 B CN 114299066B CN 202210200679 A CN202210200679 A CN 202210200679A CN 114299066 B CN114299066 B CN 114299066B
- Authority
- CN
- China
- Prior art keywords
- image
- defect
- detected
- pixel
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000007547 defect Effects 0.000 title claims abstract description 415
- 238000001514 detection method Methods 0.000 title claims abstract description 130
- 238000003709 image segmentation Methods 0.000 title claims abstract description 68
- 238000000605 extraction Methods 0.000 title claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 85
- 238000000034 method Methods 0.000 claims abstract description 64
- 230000008859 change Effects 0.000 claims abstract description 49
- 239000011521 glass Substances 0.000 claims abstract description 37
- 238000004590 computer program Methods 0.000 claims abstract description 23
- 238000012216 screening Methods 0.000 claims abstract description 19
- 238000003860 storage Methods 0.000 claims abstract description 12
- 238000004364 calculation method Methods 0.000 claims description 83
- 238000010586 diagram Methods 0.000 claims description 52
- 238000012549 training Methods 0.000 claims description 26
- 230000006870 function Effects 0.000 claims description 24
- 238000001914 filtration Methods 0.000 claims description 12
- 238000006073 displacement reaction Methods 0.000 claims description 11
- 238000004422 calculation algorithm Methods 0.000 claims description 10
- 230000011218 segmentation Effects 0.000 claims description 8
- 239000013598 vector Substances 0.000 claims description 8
- 238000005520 cutting process Methods 0.000 claims description 5
- 230000000877 morphologic effect Effects 0.000 claims description 5
- 238000010008 shearing Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 abstract description 26
- 239000000284 extract Substances 0.000 abstract description 9
- 238000007781 pre-processing Methods 0.000 abstract description 3
- 238000004519 manufacturing process Methods 0.000 description 24
- 238000010606 normalization Methods 0.000 description 17
- 230000004913 activation Effects 0.000 description 11
- 238000005516 engineering process Methods 0.000 description 6
- 230000002950 deficient Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 230000018109 developmental process Effects 0.000 description 4
- 238000012800 visualization Methods 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 3
- 238000013523 data management Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 230000003213 activating effect Effects 0.000 description 2
- 239000012535 impurity Substances 0.000 description 2
- 238000009776 industrial production Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 230000008439 repair process Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000007797 corrosion Effects 0.000 description 1
- 238000005260 corrosion Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003908 quality control method Methods 0.000 description 1
- 238000011897 real-time detection Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000011895 specific detection Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Image Analysis (AREA)
Abstract
The application relates to a method, a device, a computer device, a storage medium and a computer program product for defect detection based on salient feature pre-extraction and image segmentation. The method comprises the following steps: and acquiring an image to be detected, processing the image to be detected, segmenting the gray level change characteristic map according to an image segmentation threshold, and screening pixel points contained in the image to be detected for multiple times to obtain a target defect significance characteristic map and a defect subgraph of the image to be detected. Therefore, the data are input into a pixel point defect probability prediction model trained in advance to obtain a defect detection result image of the image to be detected. The method provided by the invention realizes the preliminary screening of the high-resolution image by preprocessing and extracting the significant characteristics of the image to be detected on the glass surface, extracts effective defect characteristics while improving the detection efficiency, assists the detection process and ensures the high fineness and the high efficiency of the detection of the defects on the glass surface.
Description
Technical Field
The application relates to the technical field of glass detection, in particular to a defect detection method and device based on saliency feature pre-extraction and image segmentation.
Background
With the rapid development of advanced manufacturing technology and the continuous improvement of the living standard of people, the demand of the society on product diversification and individuation is urgent day by day, and the requirements on the production efficiency and the quality of the product are continuously upgraded. Because the manufacturing industry has strict requirements on the surface quality of products, and any tiny defects, impurities, deformation and the like can have important influence on the aesthetic property and even the safety of the products, the effective detection of the surface quality of the products is a problem to be solved urgently.
In the related art, the product image is analyzed through an artificial intelligence algorithm, and then the glass surface defects are positioned and classified to form a novel solution. However, since the image detection algorithm in the related art has a limit in its own calculation capability, it is impossible to detect defects in a high-resolution image of the surface of a glass product.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a defect detection method, apparatus, computer device, computer readable storage medium and computer program product based on salient feature pre-extraction and image segmentation, which can accurately detect defect location points in high-resolution images.
In a first aspect, the present application provides a defect detection method based on salient feature pre-extraction and image segmentation. The method comprises the following steps:
the method comprises the steps of obtaining an image to be detected, processing the image to be detected to obtain a gray level change characteristic diagram of the image to be detected, wherein the gray level change characteristic diagram comprises pixel values of all pixel points in the image to be detected, and the image to be detected is a high-resolution image of the glass surface.
Carrying out mean value calculation on pixel values corresponding to the pixel points meeting the preset threshold value screening condition to obtain an image segmentation threshold value;
extracting a region corresponding to a pixel point of which the pixel value is greater than or equal to the image segmentation threshold value in the gray level change characteristic map to obtain a target defect significance characteristic map of the image to be detected;
according to the coordinates of the target defect saliency characteristic graph, the image to be detected is cut to obtain a defect subgraph corresponding to the target defect saliency characteristic graph;
and inputting the target defect significance characteristic graph and the defect subgraph into a pixel point defect probability prediction model trained in advance to obtain a defect detection result image of the image to be detected.
In one embodiment, the extracting a region corresponding to a pixel point whose pixel value is greater than or equal to the image segmentation threshold value in the gray-scale change feature map to obtain a target defect saliency feature map of the image to be detected includes:
determining the pixel value of a pixel point of which the pixel value is smaller than an image segmentation threshold value in the gray level change characteristic image as a target value to obtain a first defect significance characteristic image corresponding to the image to be detected;
according to a preset morphological open operation algorithm, carrying out noise point removing treatment on the first defect significance characteristic diagram to obtain a second defect significance characteristic diagram of the image to be detected;
and removing the region corresponding to the pixel point with the pixel value as the target value in the second defect significance characteristic diagram to obtain the target defect significance characteristic diagram of the image to be detected.
In one embodiment, before the step of inputting the target defect saliency feature map and the defect subgraph into a pixel defect probability prediction model trained in advance, the method further includes:
and performing image enhancement processing on the defect subgraph to obtain the processed defect subgraph, wherein the image enhancement processing comprises one or more of image random rotation processing, image displacement processing, image scaling processing, image shearing processing and image turning processing.
In one embodiment, the defect detection result image of the image to be detected includes defect probability values of a plurality of pixel points in the image to be detected, and the method further includes:
and carrying out binarization processing on the plurality of pixel points in the image to be detected according to a preset binarization segmentation threshold value and defect probability values of the plurality of pixel points in the image to be detected to obtain a defect image of the image to be detected.
In one embodiment, the method further comprises:
acquiring training data, wherein the training data comprises a sample defect significance characteristic diagram of a sample image, a sample defect subgraph and a sample defect detection result image;
inputting the sample defect significance characteristic graph and the sample defect subgraph into a pixel point defect probability prediction model to be trained to obtain a predicted defect detection result image;
calculating a loss value according to a sample defect probability value of a plurality of pixel points contained in the sample defect detection result image and a predicted defect probability value of a plurality of pixel points contained in the predicted defect detection result image by a preset loss function;
and updating the network parameters of the pixel defect probability prediction model to be trained according to the loss value, and returning to the step of executing the training data acquisition until the loss value meets the preset training completion condition to obtain the trained pixel defect probability prediction model.
In one embodiment, the inputting the sample defect significant feature map and the sample defect subgraph into a pixel point defect probability prediction model to be trained to obtain a predicted defect detection result image includes:
performing channel cascade processing on the sample defect significance characteristic graph and the sample defect subgraph to obtain a sample spliced image;
extracting the feature vectors of the sample spliced image through a first number of encoders to obtain a feature map of the sample image;
and performing convolution operation on the feature maps of the sample images through a second number of decoders to obtain the prediction defect probability value of each pixel point in the sample images, and combining to obtain the prediction defect detection result images of the sample images.
In a second aspect, the present application further provides a defect detection apparatus based on salient feature pre-extraction and image segmentation. The device comprises:
the acquisition module is used for acquiring an image to be detected and processing the image to be detected to obtain a gray level change characteristic diagram of the image to be detected, the gray level change characteristic diagram contains pixel values of all pixel points in the image to be detected, and the image to be detected is a high-resolution image of the glass surface.
The calculation module is used for carrying out mean value calculation on pixel values corresponding to the pixel points meeting the preset threshold value screening condition to obtain an image segmentation threshold value;
the extraction module is used for extracting a region corresponding to a pixel point with a pixel value larger than or equal to the image segmentation threshold value in the gray scale change characteristic map to obtain a target defect significance characteristic map of the image to be detected;
the cutting module is used for cutting the image to be detected according to the coordinates of the target defect saliency characteristic graph to obtain a defect subgraph corresponding to the target defect saliency characteristic graph;
and the detection module is used for inputting the target defect significance characteristic graph and the defect subgraph into a pixel point defect probability prediction model trained in advance to obtain a defect detection result image of the image to be detected.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the following steps when executing the computer program:
the method comprises the steps of obtaining an image to be detected, processing the image to be detected to obtain a gray level change characteristic diagram of the image to be detected, wherein the gray level change characteristic diagram comprises pixel values of all pixel points in the image to be detected, and the image to be detected is a high-resolution image of the glass surface.
Carrying out mean value calculation on pixel values corresponding to the pixel points meeting the preset threshold value screening condition to obtain an image segmentation threshold value;
extracting a region corresponding to a pixel point of which the pixel value is greater than or equal to the image segmentation threshold value in the gray level change characteristic map to obtain a target defect significance characteristic map of the image to be detected;
according to the coordinates of the target defect saliency characteristic graph, the image to be detected is cut to obtain a defect subgraph corresponding to the target defect saliency characteristic graph;
and inputting the target defect significance characteristic graph and the defect subgraph into a pixel point defect probability prediction model trained in advance to obtain a defect detection result image of the image to be detected.
In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:
the method comprises the steps of obtaining an image to be detected, processing the image to be detected to obtain a gray level change characteristic diagram of the image to be detected, wherein the gray level change characteristic diagram comprises pixel values of all pixel points in the image to be detected, and the image to be detected is a high-resolution image of the glass surface.
Carrying out mean value calculation on pixel values corresponding to the pixel points meeting the preset threshold value screening condition to obtain an image segmentation threshold value;
extracting a region corresponding to a pixel point of which the pixel value is greater than or equal to the image segmentation threshold value in the gray level change characteristic map to obtain a target defect significance characteristic map of the image to be detected;
according to the coordinates of the target defect saliency characteristic graph, the image to be detected is cut to obtain a defect subgraph corresponding to the target defect saliency characteristic graph;
and inputting the target defect significance characteristic graph and the defect subgraph into a pixel point defect probability prediction model trained in advance to obtain a defect detection result image of the image to be detected.
In a fifth aspect, the present application further provides a computer program product. The computer program product comprising a computer program which when executed by a processor performs the steps of:
the method comprises the steps of obtaining an image to be detected, processing the image to be detected to obtain a gray level change characteristic diagram of the image to be detected, wherein the gray level change characteristic diagram comprises pixel values of all pixel points in the image to be detected, and the image to be detected is a high-resolution image of the glass surface.
Carrying out mean value calculation on pixel values corresponding to the pixel points meeting the preset threshold value screening condition to obtain an image segmentation threshold value;
extracting a region corresponding to a pixel point of which the pixel value is greater than or equal to the image segmentation threshold value in the gray level change characteristic map to obtain a target defect significance characteristic map of the image to be detected;
according to the coordinates of the target defect saliency characteristic graph, the image to be detected is cut to obtain a defect subgraph corresponding to the target defect saliency characteristic graph;
and inputting the target defect significance characteristic graph and the defect subgraph into a pixel point defect probability prediction model trained in advance to obtain a defect detection result image of the image to be detected.
The defect detection method, apparatus, computer device, storage medium and computer program product based on salient feature pre-extraction and image segmentation described above, the method comprising: and acquiring an image to be detected, processing the image to be detected, segmenting the gray level change characteristic map according to an image segmentation threshold, and screening pixel points contained in the image to be detected for multiple times to obtain a target defect significance characteristic map and a defect subgraph of the image to be detected. Therefore, the data are input into a pixel point defect probability prediction model which is trained in advance, and a defect detection result image of the image to be detected is obtained. The method provided by the invention realizes the preliminary screening of the high-resolution image by preprocessing and extracting the significant characteristics of the image to be detected on the glass surface, extracts effective defect characteristics while improving the detection efficiency, assists the detection process and ensures the high fineness and the high efficiency of the detection of the defects on the glass surface.
Drawings
FIG. 1 is a schematic flow chart illustrating a defect detection method based on salient feature pre-extraction and image segmentation in one embodiment;
FIG. 2 is a statistical histogram of gray pixel values in one embodiment;
FIG. 3 is a flowchart illustrating the steps of determining a target defect saliency map in one embodiment;
FIG. 4 is a schematic flow chart of the training step in one embodiment;
FIG. 5 is a schematic flow chart diagram illustrating the steps of the model internal computation in one embodiment;
FIG. 6 is a schematic diagram of an encoder in one embodiment;
FIG. 7 is a block diagram of a decoder in one embodiment;
FIG. 8 is a flowchart illustrating a defect detection method based on salient feature pre-extraction and image segmentation in another embodiment;
FIG. 9A is a diagram illustrating an input image of a defect detection method based on salient feature pre-extraction and image segmentation in one embodiment;
FIG. 9B is a diagram illustrating an output image of a defect detection method based on salient feature pre-extraction and image segmentation, according to an embodiment;
FIG. 10 is a block diagram of a defect detection apparatus based on salient feature pre-extraction and image segmentation in one embodiment;
FIG. 11 is a block diagram of a defect detection apparatus based on salient feature pre-extraction and image segmentation in another embodiment;
FIG. 12 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
With the rapid development of advanced manufacturing technology and the continuous improvement of the living standard of people, the demand of the society on product diversification and individuation is urgent day by day, the requirements on the production efficiency and the quality of the product are continuously upgraded, and the production management mode of the traditional manufacturing industry faces huge challenges. With the development of science and technology, the Chinese manufacturing industry meets continuous development opportunities, and intelligent manufacturing becomes an important support way for the competitiveness improvement of the Chinese equipment manufacturing industry and the upgrading and reconstruction of the traditional manufacturing industry. The intelligent detection technology aims to detect the quality of a product in the manufacturing process through an intelligent algorithm, and simultaneously record and feed back data, so that powerful support is provided for the intelligent decision of subsequent links such as product sorting, repair and testing. The manufacturing industry has strict requirements on the product quality, particularly the surface quality, and any tiny defects, impurities, deformation and the like can have important influence on the product aesthetic property and even the safety, so that the effective detection on the product quality is a problem to be solved urgently. Meanwhile, the application of intelligent technology to replace traditional manual work is also a necessary trend for realizing efficient and accurate detection. Therefore, for the visual detection module in intelligent manufacturing, the core value of the visual detection module provides technical support for high-precision manufacturing industry, the intelligent detection of manufacturing defects which is driven by big data and takes an artificial intelligence technology as the core is realized, the labor force is effectively saved, and meanwhile, the production efficiency and the product quality are improved.
The surface defects of the product are directly related to the appearance of the product, and the influence on the quality of the product is self-evident. However, due to the influence of the manufacturing environment and the process flow, typical surface defects such as stains, cracks, collisions, scratches, irregular shapes and the like are frequently found on various production lines and are one of the defect types with the highest occurrence rate in the manufacturing process. The detection of the surface defects of products in the traditional manufacturing industry is usually finished manually, and some high-precision products such as glass, films and the like need technical personnel to be equipped with a professional detection device, so that the consumption of human resources is huge, and the automatic detection and the intelligent detection of the surface defects are always urgent requirements of the traditional manufacturing industry. Under the background of big data era and intelligent manufacturing, the artificial intelligence algorithm is used for analyzing the product images of glass and the like, and further positioning and classifying the surface defects become a novel solution.
The existing image detection algorithm can not realize high-efficiency and accurate detection aiming at the defects of the glass surface, and is particularly difficult when the image resolution is very high.
In an embodiment, as shown in fig. 1, a defect detection method based on salient feature pre-extraction and image segmentation is provided, and this embodiment is illustrated by applying this method to a terminal, it can be understood that this method may also be applied to a server, and may also be applied to a system including a terminal and a server, and is implemented by interaction between the terminal and the server, where the terminal may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, and the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart car-mounted devices, and the like. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device and the like, and the server can be realized by an independent server or a server cluster formed by a plurality of servers. In this embodiment, the defect detection method based on salient feature pre-extraction and image segmentation includes the following steps:
and 102, acquiring an image to be detected, and processing the image to be detected to obtain a gray level change characteristic diagram of the image to be detected.
The image to be detected comprises original pixel values of a plurality of pixel points in the image of the object to be detected, and can be represented by I; the processing of the image to be detected can be mean filtering processing of the image to be detected; the mean filtering may be two-dimensional gaussian mean filtering; the gray-scale change characteristic diagram comprises pixel values of all pixel points in the image to be detected after mean value filtering, the pixel values are pixel gray values, the object to be detected can be glass, and the image to be detected can be a high-resolution image of the surface of the glass.
In this embodiment, the terminal acquires an image to be detected of an object to be detected, and performs two-dimensional gaussian mean filtering on a pixel value of each pixel included in the image to be detected to obtain a mean filtered image. In this way, the terminal performs difference processing on the image to be detected and the mean value filtering image, and calculates the gray level change characteristic diagram of the image to be detected. In an actual application scenario, the gray scale change characteristic map corresponding to the detected image can be calculated by the following formula:
F=I-G(I),
wherein,Fis a characteristic diagram of the gray-scale variation,Iin order to detect the image to be detected,Gfor mean filtering (e.g., two-dimensional gaussian filtering),G (I)the method is to obtain a mean value filtering image after mean value filtering is carried out on an image to be detected.
And 104, performing mean value calculation on pixel values corresponding to the pixel points meeting the preset threshold value screening condition to obtain an image segmentation threshold value.
The preset threshold screening condition may be that, within pixel values of a plurality of pixels included in the image to be detected, a first number of pixels having a largest pixel value, for example, the first number may be ten percent of the number of pixels included in the image to be detected, the number of pixels included in the image to be detected may be 100, and then the corresponding first number may be 10.
In this embodiment, the terminal screens each pixel point included in the gray-scale change characteristic map according to a preset threshold screening condition, and extracts a first number of pixel points that satisfy the preset threshold screening condition. In this way, the terminal may perform an average processing on the first number of pixel values of the first number of pixel points to obtain an image segmentation threshold.
In one example, the terminal counts the pixel points of which the pixel values are within 0-255 in the gray scale change characteristic image, and the statistical result is a gray scale histogram which can represent the defect significance characteristics of the image to be detected. The histogram in a specific implementation may be as shown in fig. 2, where an abscissa (Intensity) of the histogram represents a pixel gray-scale value, and an ordinate (N) represents the number of pixels. The terminal can extract the first ten percent of pixels with the maximum pixel gray value from the pixel point set with the pixel gray value not being zero in the gray variation characteristic image according to the statistical information represented by the gray histogram, the pixels serve as the pixels meeting the preset threshold screening condition, the pixel values of the pixels meeting the preset threshold screening condition are subjected to mean value calculation, and the image segmentation threshold is determined.
And 106, extracting the corresponding region of the pixel point with the pixel value larger than or equal to the image segmentation threshold value in the gray level change characteristic map to obtain a target defect significance characteristic map of the image to be detected.
Specifically, the terminal can segment the gray-scale change feature map based on the image segmentation threshold to obtain a target defect significance feature map of the image to be detected. The specific segmentation process may be: and traversing the pixel values of all the pixel points contained in the gray scale change characteristic map by the terminal, and setting the pixel values of the pixel points with the pixel values smaller than the image segmentation threshold value to be zero (zero gray value) to obtain a first defect significance characteristic map. In this way, the terminal may traverse the pixel values of the pixels included in the first defect saliency feature map again, extract the regions corresponding to the pixels whose pixel values are not zero, or extract the regions corresponding to the pixels whose pixel values are greater than or equal to the image segmentation threshold, to obtain the target defect saliency feature map of the image to be detected.
And 108, cutting the image to be detected according to the coordinates of the target defect saliency characteristic graph to obtain a defect subgraph corresponding to the target defect saliency characteristic graph.
Specifically, the terminal determines a position to be cut on the image to be detected according to coordinates of each pixel point contained in the target defect significance characteristic diagram in the image to be detected, and cuts the image to be detected according to the position to be cut to obtain a defect subgraph corresponding to the target defect significance characteristic diagram. The defect subgraph is a partial image in which defects may exist in the image to be detected. In the case where the image to be detected is a high-resolution glass surface image, the defect sub-image represents a partial image of the high-resolution glass surface image that may contain defects.
And 110, inputting the target defect significance characteristic graph and the defect subgraph into a pixel point defect probability prediction model trained in advance to obtain a defect detection result image of the image to be detected.
Specifically, the pixel defect probability prediction model trained in advance is used for predicting the defect probability of a plurality of pixels contained in the image to be detected according to the target defect significance characteristic graph and the defect subgraph to obtain the defect probability of each pixel contained in the image to be detected. Therefore, the terminal can take the image containing the defect probability of each pixel point as a defect detection result image of the image to be detected.
In the defect detection method based on the salient feature pre-extraction and the image segmentation, an image to be detected is obtained and processed, a gray change feature map is segmented according to an image segmentation threshold, and pixel points contained in the image to be detected are screened for multiple times to obtain a target defect salient feature map and a defect subgraph of the image to be detected. Therefore, the data are input into a pixel point defect probability prediction model which is trained in advance, and a defect detection result image of the image to be detected is obtained. The defect detection method provided by the invention realizes the preliminary screening of the high-resolution image by preprocessing the image to be detected and extracting the significant characteristics, improves the detection efficiency, extracts the effective defect characteristics, assists the detection process and ensures the high fineness and the high efficiency of the detection of the defects on the surface of the glass. Therefore, the method provided by the invention can classify the product image pixel by pixel while ensuring the detection time efficiency, and accurately segment the defect area to meet the requirement of industrial automatic detection. The high-precision defect positioning can meet the quality detection requirement of high-quality products, and has great significance for subsequent links of product quality control, intelligent sorting, product repair and the like in the intelligent manufacturing process.
In one embodiment, as shown in fig. 3, the specific processing procedure of step 108 "extracting a region corresponding to a pixel point whose pixel value is greater than or equal to an image segmentation threshold value in a gray-scale change feature map to obtain a target defect saliency feature map of an image to be detected" includes:
step 202, determining the pixel value of the pixel point with the pixel value smaller than the image segmentation threshold value in the gray scale change characteristic image as a target value, and obtaining a first defect significance characteristic image corresponding to the image to be detected.
The target value may be used to distinguish whether the pixel value of the pixel point is greater than or equal to the image segmentation threshold or whether the pixel value of the pixel point is smaller than the image segmentation threshold.
Specifically, the terminal traverses pixel values of all pixel points included in the gray scale change feature map, and sets the pixel value of the pixel point of which the pixel value is smaller than the image segmentation threshold as a target value to obtain a first defect saliency feature map. For example, the target value may be zero (zero gray value). In this way, the pixel value of each pixel included in the first defect saliency map includes the target value, is greater than the image segmentation threshold, and is equal to the image segmentation threshold.
And 204, performing noise point removal processing on the first defect significance characteristic diagram according to a preset morphological open operation algorithm to obtain a second defect significance characteristic diagram of the image to be detected.
Specifically, for the first defect saliency feature map, the terminal may perform noise removal processing, so as to further improve accuracy and precision of image detection. The specific process of the terminal performing noise removal processing may include: and carrying out corrosion treatment and expansion treatment on the first defect significance characteristic diagram through a preset morphological open operation algorithm to obtain a second defect significance characteristic diagram.
And step 206, removing the area corresponding to the pixel point with the pixel value as the target value in the second defect saliency characteristic map to obtain the target defect saliency characteristic map of the image to be detected.
Specifically, the terminal may traverse the pixel values of the pixel points included in the second defect saliency feature map again, extract the regions corresponding to the pixel points whose pixel values are not zero, that is, extract the regions corresponding to the pixel points whose pixel values are greater than or equal to the image segmentation threshold, and obtain the target defect saliency feature map of the image to be detected.
In this embodiment, the target defect saliency characteristic map is obtained by removing the gray level change characteristic map, and the accuracy and precision of image detection can be improved on the premise of ensuring the detection efficiency.
In one embodiment, before the step 110 "inputting the target defect saliency feature map and the defect subgraph into the pixel defect probability prediction model trained in advance", the defect detection method based on the saliency feature pre-extraction and the image segmentation further includes:
and carrying out image enhancement processing on the defect subgraph to obtain the processed defect subgraph.
The image enhancement processing comprises one or more of image random rotation processing, image displacement processing, image scaling processing, image shearing processing and image turning processing, and the processing modes do not distinguish the execution sequence.
In an example, the process of randomly rotating the image of the defect subgraph by the terminal may be that the terminal determines a target rotation angle according to a random strategy in an angle set, the angle set includes all angle values between 0 degree and 360 degrees, and the target rotation angle is any angle value between 0 degree and 360 degrees. Therefore, the terminal can rotate the defect subgraph by the target rotation angle to obtain the processed defect subgraph.
The process of the terminal performing the image displacement processing on the defect subgraph may be that the terminal first obtains the size information of the defect subgraph, including the length value in the horizontal direction and the length value in the vertical direction, so that the terminal can calculate the displacement distance according to the size information, and then perform the displacement processing on the defect subgraph in the horizontal direction or the vertical direction according to the displacement distance. The displacement distance may be a ten percent horizontal length value or a ten percent vertical length value. For example, the horizontal displacement distance or the vertical displacement distance is in the range of 0% to 10% of the image size.
The process of the terminal performing image scaling on the defect subgraph may be that the terminal first obtains size information of the defect subgraph, including a length value in a horizontal direction and a length value in a vertical direction, so that the terminal may perform enlargement processing or reduction processing on the defect subgraph. The terminal can perform amplification processing or reduction processing on the defect subgraph within a preset range, wherein the preset range is 0% to 5% of size information of the defect subgraph.
The process of the terminal performing the image clipping processing on the defect subgraph may be that the terminal first obtains the size information of the defect subgraph, including the length value in the horizontal direction and the length value in the vertical direction, so that the terminal may clip the defect subgraph. The terminal can cut the defect subgraph within a preset range, wherein the preset range is 0% to 5% of the size information of the defect subgraph.
The process of the terminal for turning over the image of the defect subgraph can be that the terminal turns over the defect subgraph in the horizontal direction or in the vertical direction according to a preset random strategy. In another example, the terminal may randomly perform horizontal translation or vertical translation within a certain coordinate range on the defect subgraph; randomly rotating an input image within a certain angle range; carrying out random zooming within a certain magnification range on an input image; the input image is randomly flipped horizontally or vertically.
In this embodiment, by performing image enhancement processing on the defective subgraph, parameters of the image to be detected can be further adjusted (reduced), and the image detection efficiency is improved.
In one embodiment, the defect detection result image of the image to be detected comprises defect probability values of a plurality of pixel points in the image to be detected. Correspondingly, the defect detection method based on the salient feature pre-extraction and the image segmentation further comprises the following steps:
and carrying out binarization processing on the plurality of pixel points in the image to be detected according to a preset binarization segmentation threshold value and defect probability values of the plurality of pixel points in the image to be detected to obtain a defect image of the image to be detected.
Specifically, the pre-trained pixel defect probability prediction model is used for predicting the defect probability of a plurality of pixels contained in the image to be detected according to the target defect significance characteristic diagram and the defect subgraph to obtain the defect probability of each pixel contained in the image to be detected and obtain a defect detection result image of the image to be detected. Therefore, the terminal can carry out binarization processing according to the defect probability of each pixel point contained in the defect detection result image of the image to be detected and a preset probability threshold value to obtain the defect image of the image to be detected. The specific process of binarization may be: and the terminal carries out condition judgment on the defect probability of each pixel point, takes the pixel points with the defect probability larger than or equal to a preset probability threshold value as defect pixel points, and takes the pixel points with the defect probability smaller than the preset probability threshold value as normal pixel points. Therefore, the terminal can take the image formed by each defective pixel point as the defect detection result image of the image to be detected.
In an example, the preset probability threshold may be 0.5, or other parameter values determined according to an actual application scenario, or a preset probability threshold obtained in response to an input operation of a user in the input operation.
In one embodiment, as shown in fig. 4, the defect detection method based on salient feature pre-extraction and image segmentation further includes:
step 302, training data is obtained.
The training data comprises a sample defect significance characteristic graph of a sample image, a sample defect subgraph and a sample defect detection result image. The training data includes a plurality of sample images, which may be, for example, a plurality of grayscale images of the high-resolution glass surface, a sample defect subgraph of each grayscale image, and a sample defect detection result image thereof.
And 304, inputting the sample defect significance characteristic graph and the sample defect subgraph into a pixel point defect probability prediction model to be trained to obtain a predicted defect detection result image.
The pixel defect probability prediction model to be trained can be a semantic segmentation model based on deep learning, and the model comprises an encoder module and a decoder module. The encoder module can comprise a plurality of encoders, and each encoder comprises a stride convolution layer, a Batch Normalization layer and an activation function layer; similarly, the decoder module may include a plurality of channel cascade units and a plurality of decoders, each of the decoders includes a convolutional layer, a Batch Normalization layer, an activation function layer, and an upsampling layer, an upsampling method used in the upsampling layer may be bilinear interpolation, an input size of each encoder is different from an input size, and an input size of each decoder is different from an input size.
Specifically, a terminal acquires a sample image in training data, processes the sample image to obtain a sample defect saliency feature map and a sample defect subgraph corresponding to the sample image, performs channel cascade processing on the sample image, inputs the obtained multichannel image with defect features into a first encoder in an encoder module, calculates the feature map according to the multichannel image with defect features through learnable convolution kernel in the first encoder, and each pixel position in the feature map is represented by a feature vector; for the ith encoder, the input is a profile of the output of the (i-1) th encoder.
The terminal inputs the feature map output by the last encoder into a first decoder in a decoder module, the output of the first decoder is the decoded feature map, the input of the ith decoder is the output of the (i-1) th decoder and the result (spliced according to the feature channels) of the feature map of the corresponding scale encoder after channel cascade processing, the input of the last decoder is the output of the previous decoder, and the output of the last decoder is the predicted defect probability of each pixel point position of the sample image, namely the single-channel score map.
And step 306, calculating a loss value according to the sample defect probability values of a plurality of pixel points contained in the sample defect detection result image and the predicted defect probability values of a plurality of pixel points contained in the predicted defect detection result image by presetting a loss function.
Specifically, the loss value may be calculated according to a preset loss function, a sample defect probability value of a plurality of pixel points, and a predicted defect probability value of a plurality of pixel points included in the predicted defect detection result image, where the preset loss function may be a local loss function, and specifically, the loss value may be calculated by the following formulaLoss:
Wherein,nrepresenting the total number of pixel points,p i is shown asiThe probability value of the predicted defect of each pixel point,y i is shown asiThe sample defect probability values of the individual pixel points (the sample defect probability value at the defect point position is 1, the sample defect probability value at the non-defect point position is 0),logis represented by a natural numbereA base logarithm operation;αandγis a constant parameter, set toα=0.75,γ=2。
And 308, updating the network parameters of the pixel defect probability prediction model to be trained according to the loss value, and returning to the step of acquiring training data until the loss value meets the preset training completion condition to obtain the trained pixel defect probability prediction model.
Wherein the preset training completion condition may be a lossThe loss function corresponding to the value has converged, or the number of iterations of the training data has reached the target number, etc. For example, the target number of times may be 1 × 105~2×105And the like, the target times are not particularly limited in the embodiment of the present invention.
Specifically, according to the loss value, calculating new network parameters of the pixel defect probability prediction model to be trained, and then updating the pixel defect probability prediction model to be trained to obtain an updated pixel defect probability prediction model. Then, the terminal re-inputs the training data into the updated pixel defect probability prediction model, and re-executes the steps of the method of the embodiment until the calculated loss value meets the preset training completion condition, so as to obtain the trained pixel defect probability prediction model.
Optionally, the input of the pixel defect probability prediction model to be trained is a gray image of the surface of the high-resolution glass, and the data type is a single-channel matrix of uint 8; and outputting a glass surface subimage obtained by preliminary positioning and a corresponding defect significance characteristic image, namely a defect image of the image to be detected.
In this embodiment, through the setting of the model and the training process of the model, the recognition performance of the model can be improved and the calculation amount of training and the calculation amount of recognition can be reduced on the premise of ensuring the accuracy of feature extraction.
In an embodiment, as shown in fig. 5, a specific processing procedure of step 304 "inputting the sample defect saliency feature map and the sample defect subgraph into a pixel defect probability prediction model to be trained to obtain a predicted defect detection result image" includes:
and step 402, performing channel cascade processing on the sample defect significance characteristic graph and the sample defect subgraph to obtain a sample spliced image.
Specifically, the pixel defect probability prediction model to be trained may include a first channel cascade module, where the first channel cascade module is configured to receive a sample defect significance feature map and a sample defect subgraph input by a terminal, perform channel cascade on the two images, and obtain a sample mosaicAnd (6) receiving the image. For example, a channel cascade refers to two would have a width of: (w) High, high (h) A channel (a)c) The images of 3 dimensions are stitched according to the channel dimensions.
In one example, the sample defect subgraph can be a defect subgraph of a high resolution glass surface image, and the size information can be 224 × 224 × 1 (width × height × number of channels); the size information of the sample defect saliency map may be 224 × 224 × 1 (width × height × number of channels). In this way, the terminal performs channel cascade on the two images through the first channel cascade module to obtain 1 image of 224 × 224 × 2 (width × height × number of channels).
And step 404, performing feature vector extraction processing on the sample spliced image through a first number of encoders to obtain a feature map of the sample image.
Wherein, the structure diagram of each encoder may be as shown in fig. 6, and includes a first input subunit(s) ((Input features) A first convolution calculation subunit, a second convolution calculation subunit, a third convolution calculation subunit and a first output subunit (Out features). The first convolution calculation subunit comprises a 1 × 1 convolution (1 × 1)Conv) Layer, normalization: (BN) Layer and activation function: (ReLU) Layer, second convolution calculation subunit comprising a 3 × 3 packet convolution (3 × 3)Group Conv) A layer, a normalization layer, and an activation function layer, and a third convolution calculating subunit comprising a 1 × 1 convolution (1 × 1)Conv) Layer, normalization layer. Wherein, 3 × 3 packet convolution (3 × 3)Group Conv) The layers are grouped convolution calculations with convolution kernels of 3 x 3 size, and the normalization layer is obtained byBatch NormalizationThe function is normalized, 1 × 1 convolution (1 × 1)Conv) The layers are convolution calculated with convolution kernel of 1 × 1 size and the activation function layer is formed byReLUThe function performs activation processing.
Specifically, the terminal inputs the sample-stitched image first into the first input subunit in the first encoder. In the first encoder, a terminal firstly calculates a sample spliced image through a first convolution calculation subunit, the first convolution calculation subunit inputs a calculation result to a second convolution calculation subunit, the second convolution calculation subunit calculates the sample spliced image, and inputs the calculation result to a third convolution calculation subunit to obtain a first convolution calculation result. And the terminal takes the sum of the first convolution calculation result and the sample splicing image as an output result of the first encoder.
In one example, the calculation of the sum of the first convolution calculation result and the sample stitched image may be: and adding the first convolution calculation result and the sample splicing image according to the corresponding channel position to obtain a sum. The calculation process of the other encoders is similar to that of the first encoder, and is not described herein.
In one example, the first input subunit inputs the sample stitched image to a first convolution calculation subunit where the sample stitched image would first be convolved by 1 × 1 (1 × 1)Conv) Layer, carrying out convolution calculation with convolution kernel of 1 × 1 size to obtain a first initial convolution calculation result; the initial convolution calculation result is input into the normalization layer according toBatch NormalizationNormalizing the function to obtain a normalized result, inputting the normalized result into the activation function layer, and performing normalization according to the normalized resultReLUAnd activating the normalization result by the function to obtain a second initial convolution calculation result. That is, the output of the first convolution operator unit is the second initial convolution calculation result.
In this way, the terminal inputs the second initial convolution calculation result to the second convolution calculation subunit, and the second initial convolution calculation result is sequentially convoluted by 3 × 3 packets (3 × 3)Group Conv) And the convolution layer, the normalization layer and the activation function layer obtain a third initial convolution calculation result. And the terminal inputs the third initial convolution calculation result into a third convolution calculation subunit, and sequentially performs convolution calculation and normalization processing on the 1 × 1 convolution layer on the third initial convolution calculation result to obtain a first convolution calculation result. And the terminal takes the sum of the first convolution calculation result and the sample splicing image as an output result of the first encoder.
And 406, performing convolution operation on the feature map of the sample image through a second number of decoders to obtain the predicted defect probability value of each pixel point in the sample image, and combining to obtain a predicted defect detection result image of the sample image.
Wherein, the structure diagram of each decoder may be as shown in fig. 7, and includes a second input subunit(s) ((s))Input features) A fourth convolution calculation subunit, a fifth convolution calculation subunit, a sixth convolution calculation subunit, and an upsampling subunit (a)UpSampling) And a second output subunit (Out features). The fourth convolution calculation subunit comprises a 3 × 3 packet convolution (3 × 3)Group Conv) Layer, normalization: (BN) Layer and activation function: (ReLU) Layer, 3 x 3 packet convolution (3 x 3)Group Conv) The layers are grouped convolution calculations with convolution kernels of 3 x 3 size, and the normalization layer is obtained byBatch NormalizationThe function is normalized by activating the function layerReLUThe function performs activation processing. The fifth convolution calculation subunit includes a 1 × 1 convolution (1 × 1)Conv) Layer, normalization layer and activation function layer, 1 × 1 convolution (1 × 1)Conv) The layer is subjected to convolution calculation with a convolution kernel of 1 × 1 size, and the fifth convolution calculation subunit has the same structure as the sixth convolution subunit.
Specifically, the terminal inputs the feature map of the sample image output by the encoder module into a first decoder, in the first decoder, the feature map of the sample image passes through a fourth convolution calculation subunit and a fifth convolution calculation subunit in sequence to obtain a second convolution calculation result, and the feature map of the sample image is also directly input into a sixth convolution calculation subunit to obtain a third convolution calculation result. And the terminal inputs the sum of the third convolution calculation result and the second convolution calculation result to the upper sampling subunit to obtain an output result of the first decoder.
Optionally, the calculation process of the sum of the first convolution calculation result and the second convolution calculation result may be: and adding the two convolution calculation results according to the corresponding channel positions to obtain a sum. The calculation process of the other decoders is similar to that of the first decoder, and is not described herein again. The specific calculation processes of the fourth convolution calculation subunit, the fifth convolution calculation subunit and the sixth convolution calculation subunit are similar to the calculation processes of the first convolution calculation subunit, the second convolution calculation subunit and the third convolution calculation subunit, and are not described herein again.
In this embodiment, the input of each encoder is a defect region sub-graph (sample defect significance feature graph and sample defect sub-graph) or a feature graph output by the last encoder, and the output is a feature graph (each pixel position is represented by a feature vector) calculated by a learnable convolution kernel. The encoder performs layer-by-layer feature extraction on the original image to generate a layer-by-layer shallow-to-deep feature map. Each decoder decodes the deep features by convolution operations to predict the probability of each pixel location being a defect while restoring the shallow features. The cascade operation is to make the decoder receive coding features of different levels and different scales, so as to improve the accuracy of the detail of the result.
The actual using process of the trained pixel defect probability prediction model may be described in detail below with reference to a specific embodiment, as shown in fig. 8:
the terminal can apply the pixel point defect probability prediction model to the actual application scene of the glass product surface defect detection. The specific detection process may be: performing the processes of step 102 to step 108 on an image to be detected (a high-resolution glass surface image) to perform feature pre-extraction and defect clipping processing to obtain a glass surface subgraph (a defect subgraph) which possibly contains defects and a defect feature map (a target defect significance feature map) corresponding to the subgraph; the terminal carries out channel cascade processing on the obtained glass surface subgraph (defect subgraph) which possibly contains defects and a corresponding defect feature map (target defect significance feature map) to obtain a multi-channel image with defect features, the multi-channel image (spliced image) is input to a trained pixel point defect probability prediction model to obtain a defect detection result image (single-channel score map) of the image to be detected, and the result image comprises defect probability values of a plurality of pixel points in the detected image. The terminal can carry out binarization processing on a plurality of pixel points in the image to be detected based on a preset binarization segmentation threshold value and defect probability values of the plurality of pixel points in the image to be detected, so as to obtain a defect image (defect segmentation result) of the image to be detected.
The high resolution glass side image is shown in FIG. 9A and the defect image of the image to be detected is shown in FIG. 9B. For convenient display, a large-area non-defective area is cut out, an intermediate process is omitted, and a display result is a final result obtained by restoring the detection result of the subgraph to the corresponding position of the original graph. As can be seen from the results, the detection model can segment the fine and complex defects on the glass surface very accurately. Meanwhile, in the detection process, the high-resolution image is cut through simple feature pre-extraction operation, so that the image size of an input detection model is greatly reduced, and the convolution operation amount of a neural network is also greatly reduced, for example, the input image size is 7168 × 498 and is about 3.6 × 106Pixel, and the extracted defect subgraph is about 256 × 256, about 6.5 × 104The number of pixels and subgraphs is about 10, and the number of image pixels is in direct proportion to the amount of convolution calculation, so that the amount of detection calculation is reduced by about 5 times, and the amount of calculation required by feature pre-extraction is small and almost negligible. In addition, the surface defects of the glass products are usually very small in area, the defect area ratio is smaller, the calculation amount saved by the method is larger, the detection efficiency is obviously improved, the real-time detection requirement of an industrial production line is met, and the method has very high practical value.
In an example, the trained pixel defect probability prediction model may include an encoder and a decoder, the encoder module may include a plurality of encoders, the decoder module may include a plurality of decoders and a plurality of channel cascade units, and specific parameter information of each encoder, each decoder, and each channel cascade unit may be as shown in table 1 below:
TABLE 1
It should be understood that, although the steps in the flowcharts related to the embodiments described above are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.
Based on the same inventive concept, the embodiment of the present application further provides a defect detection apparatus based on salient feature pre-extraction and image segmentation, for implementing the above-mentioned defect detection method based on salient feature pre-extraction and image segmentation. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the above method, so that the specific limitations in one or more embodiments of the defect detection apparatus based on salient feature pre-extraction and image segmentation provided below can be referred to the limitations in the above defect detection method based on salient feature pre-extraction and image segmentation, and are not described herein again.
In one embodiment, as shown in fig. 10, there is provided a defect detecting apparatus 600 based on salient feature pre-extraction and image segmentation, including: an obtaining module 601, a calculating module 602, an extracting module 603, a clipping module 604 and a detecting module 605, wherein:
the acquiring module 601 is configured to acquire an image to be detected, and process the image to be detected to obtain a gray level change feature map of the image to be detected, where the gray level change feature map includes pixel values of each pixel point in the image to be detected;
the calculating module 602 is configured to perform mean calculation on pixel values corresponding to pixel points meeting a preset threshold screening condition to obtain an image segmentation threshold;
an extracting module 603, configured to extract a region corresponding to a pixel point whose pixel value is greater than or equal to an image segmentation threshold in the gray-scale change feature map, to obtain a target defect saliency feature map of the image to be detected;
the cropping module 604 is configured to crop an image to be detected according to the coordinates of the target defect saliency feature map to obtain a defect sub-map corresponding to the target defect saliency feature map;
the detection module 605 is configured to input the target defect saliency feature map and the defect subgraph into a pixel defect probability prediction model trained in advance, so as to obtain a defect detection result image of the image to be detected.
In one embodiment, the extraction module is specifically configured to:
determining the pixel value of a pixel point of which the pixel value is smaller than an image segmentation threshold value in the gray level change characteristic image as a target value to obtain a first defect significance characteristic image corresponding to the image to be detected;
according to a preset morphological open operation algorithm, carrying out noise point removal processing on the first defect significance characteristic diagram to obtain a second defect significance characteristic diagram of the image to be detected;
and removing the region corresponding to the pixel point with the pixel value as the target value in the second defect significance characteristic diagram to obtain the target defect significance characteristic diagram of the image to be detected.
In one embodiment, the apparatus further comprises:
and the image processing module is used for carrying out image enhancement processing on the defect subgraph to obtain the processed defect subgraph, wherein the image enhancement processing comprises one or more of image random rotation processing, image displacement processing, image scaling processing, image shearing processing and image overturning processing.
In one embodiment, the apparatus further comprises:
and the binarization processing module is used for carrying out binarization processing on a plurality of pixel points in the image to be detected according to a preset binarization segmentation threshold value and defect probability values of the plurality of pixel points in the image to be detected to obtain a defect image of the image to be detected.
In one embodiment, the apparatus further comprises:
the training data acquisition module is used for acquiring training data, and the training data comprises a sample defect significance characteristic diagram of a sample image, a sample defect subgraph and a sample defect detection result image;
the input module is used for inputting the sample defect significance characteristic graph and the sample defect subgraph into a pixel point defect probability prediction model to be trained to obtain a predicted defect detection result image;
the loss value calculation module is used for calculating a loss value according to the sample defect probability values of a plurality of pixel points contained in the sample defect detection result image and the predicted defect probability values of a plurality of pixel points contained in the predicted defect detection result image through a preset loss function;
and the updating module is used for updating the network parameters of the pixel defect probability prediction model to be trained according to the loss value and returning to the step of acquiring the training data until the loss value meets the preset training completion condition to obtain the trained pixel defect probability prediction model.
In one embodiment, the input module is specifically configured to:
performing channel cascade processing on the sample defect significance characteristic graph and the sample defect subgraph to obtain a sample spliced image;
performing feature vector extraction processing on the sample spliced image through a first number of encoders to obtain a feature map of the sample image;
and performing convolution operation on the characteristic graph of the sample image through a second number of decoders to obtain the predicted defect probability value of each pixel point in the sample image, and combining to obtain the predicted defect detection result image of the sample image.
The modules in the defect detection apparatus based on salient feature pre-extraction and image segmentation may be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In another embodiment, the present invention further provides a glass defect detecting apparatus 1000 for pre-extracting salient features and segmenting an image, which is described in detail below with reference to fig. 11. The device comprises the following components: the image acquisition device 100, the image processing and detection device 200, the defect visualization device 400, the defect alarm device 300 and the data storage and management device 500.
The image acquisition device 100 is used for acquiring surface images of glass products by adopting an industrial linear array camera, comprises a conveyor belt, a light source and the industrial linear array camera and outputs high-resolution glass product surface gray level images; the image processing and detecting device 200 uses the image collected by the image collecting device 100 as input through the program instructionGPUAndCPUperforming feature pre-extraction related calculation processing on the image, then performing defect detection calculation, and finally outputting a defect detection result; the defect detection result will be delivered to the defect visualization device 400 and the data storage and management device 500. The defect visualization device is used for visually displaying the defect detection result, including the position, type and shape of the defect. The visualization device 400 is controlled by the detection device 200, usingLEDThe liquid crystal display completes the display. The data storage and management device 500 stores and manages the detection output result, and can realize reading and query at any time. The device is controlled by a computer program, and a storage medium is a mechanical hard disk; the defect detection result is input into the defect alarm device 300 through a decision (alarm or no alarm) to give an alarm when defective substandard products appear in the glass industrial production line. The alarm device 300 is also controlled by the detection device 200, and the output is divided into two forms of audio and light, which are respectively completed by using a loudspeaker and an LED alarm lamp.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 12. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing the detection data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a defect detection method based on salient feature pre-extraction and image segmentation.
Those skilled in the art will appreciate that the architecture shown in fig. 12 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of the above method embodiments.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, carries out the steps in the method embodiments described above.
It will be understood by those skilled in the art that all or part of the flow of the method for implementing the above embodiments may be instructed by a computer programAssociated hardware, the computer program may be stored in a non-volatile computer readable storage medium, and when executed, the computer program may include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The non-volatile memory may include a read-only memory (R-ROM)Read-Only Memory,ROM) Magnetic tape, floppy disk, flash memory, optical memory, high-density embedded nonvolatile memory, and resistance change memoryReRAM) Magnetic change memory (1)Magnetoresistive Random Access Memory,MRAM) Ferroelectric memory device (Ferroelectric Random Access Memory,FRAM) Phase change memory devicePhase Change Memory, PCM) Graphene memory, etc. The volatile memory may comprise random access memory (Random Access Memory, RAM) Or external cache memory, etc. By way of illustration and not limitation,RAMcan be in various forms, such as static random access memory (Static Random Access Memory,SRAM) Or dynamic random access memory (Dynamic Random Access Memory,DRAM) And the like. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.
Claims (9)
1. A defect detection method based on salient feature pre-extraction and image segmentation is characterized by comprising the following steps:
acquiring an image to be detected to obtain a gray level change characteristic map of the image to be detected, wherein the gray level change characteristic map comprises pixel values obtained after mean value filtering of all pixel points in the image to be detected, and the image to be detected is a high-resolution image of the glass surface;
carrying out mean value calculation on pixel values corresponding to the pixel points meeting the preset threshold value screening condition to obtain an image segmentation threshold value;
extracting a region corresponding to a pixel point of which the pixel value is greater than or equal to the image segmentation threshold value in the gray level change characteristic map to obtain a target defect significance characteristic map of the image to be detected;
according to the coordinates of the target defect saliency characteristic graph, the image to be detected is cut to obtain a defect subgraph corresponding to the target defect saliency characteristic graph;
inputting the target defect significance characteristic graph and the defect subgraph into a pixel point defect probability prediction model trained in advance to obtain a defect detection result image of the image to be detected;
inputting the target defect significance characteristic graph and the defect subgraph into a pixel point defect probability prediction model trained in advance to obtain a defect detection result image of the image to be detected, wherein the defect detection result image comprises the following steps:
performing channel cascade processing on the target defect significance characteristic graph and the defect subgraph to obtain a spliced image;
extracting the feature vectors of the spliced images through a first number of encoders to obtain a feature map of the image to be detected;
and performing convolution operation on the characteristic diagram of the image to be detected through a second number of decoders to obtain the defect probability value of each pixel point in the image to be detected, and combining to obtain the defect detection result image of the image to be detected.
2. The method according to claim 1, wherein the extracting a region corresponding to a pixel point with a pixel value greater than or equal to the image segmentation threshold value from the gray-scale change feature map to obtain a target defect saliency feature map of the image to be detected comprises:
determining the pixel value of a pixel point of which the pixel value is smaller than an image segmentation threshold value in the gray level change characteristic image as a target value to obtain a first defect significance characteristic image corresponding to the image to be detected;
according to a preset morphological open operation algorithm, carrying out noise point removal processing on the first defect significance characteristic diagram to obtain a second defect significance characteristic diagram of the image to be detected;
and removing the region corresponding to the pixel point with the pixel value as the target value in the second defect significance characteristic diagram to obtain the target defect significance characteristic diagram of the image to be detected.
3. The method of claim 1, wherein before the step of inputting the target defect saliency feature map and the defect sub-map into a pre-trained pixel point defect probability prediction model, the method further comprises:
and performing image enhancement processing on the defect subgraph to obtain the processed defect subgraph, wherein the image enhancement processing comprises one or more of image random rotation processing, image displacement processing, image scaling processing, image shearing processing and image turning processing.
4. The method of claim 1, wherein the defect detection result image of the image to be detected comprises defect probability values of a plurality of pixels in the image to be detected, and the method further comprises:
and carrying out binarization processing on the plurality of pixel points in the image to be detected according to a preset binarization segmentation threshold value and defect probability values of the plurality of pixel points in the image to be detected to obtain a defect image of the image to be detected.
5. The method of claim 1, further comprising:
acquiring training data, wherein the training data comprises a sample defect significance characteristic diagram of a sample image, a sample defect subgraph and a sample defect detection result image;
inputting the sample defect significance characteristic graph and the sample defect subgraph into a pixel point defect probability prediction model to be trained to obtain a predicted defect detection result image;
calculating a loss value according to a sample defect probability value of a plurality of pixel points contained in the sample defect detection result image and a predicted defect probability value of a plurality of pixel points contained in the predicted defect detection result image by a preset loss function;
and updating the network parameters of the pixel defect probability prediction model to be trained according to the loss value, and returning to the step of executing the training data acquisition until the loss value meets the preset training completion condition to obtain the trained pixel defect probability prediction model.
6. The method of claim 5, wherein the step of inputting the sample defect significance feature map and the sample defect subgraph into a pixel defect probability prediction model to be trained to obtain a predicted defect detection result image comprises:
performing channel cascade processing on the sample defect significance characteristic graph and the sample defect subgraph to obtain a sample spliced image;
extracting the feature vectors of the sample spliced image through a first number of encoders to obtain a feature map of the sample image;
and performing convolution operation on the feature maps of the sample images through a second number of decoders to obtain the prediction defect probability value of each pixel point in the sample images, and combining to obtain the prediction defect detection result images of the sample images.
7. A defect detection device based on salient feature pre-extraction and image segmentation, the device comprising:
the device comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring an image to be detected to obtain a gray change characteristic diagram of the image to be detected, the gray change characteristic diagram comprises pixel values obtained after mean filtering of all pixel points in the image to be detected, and the image to be detected is a high-resolution image of the glass surface;
the computing module is used for carrying out mean value computation on pixel values corresponding to the pixel points which accord with the preset threshold value screening condition to obtain an image segmentation threshold value;
the extraction module is used for extracting a region corresponding to a pixel point with a pixel value larger than or equal to the image segmentation threshold value in the gray scale change characteristic map to obtain a target defect significance characteristic map of the image to be detected;
the cutting module is used for cutting the image to be detected according to the coordinates of the target defect saliency characteristic graph to obtain a defect subgraph corresponding to the target defect saliency characteristic graph;
the detection module is used for inputting the target defect significance characteristic graph and the defect subgraph into a pixel point defect probability prediction model trained in advance to obtain a defect detection result image of the image to be detected;
the detection module is specifically configured to: performing channel cascade processing on the target defect significance characteristic graph and the defect subgraph to obtain a spliced image;
extracting the feature vectors of the spliced images through a first number of encoders to obtain a feature map of the image to be detected;
and performing convolution operation on the characteristic diagram of the image to be detected through a second number of decoders to obtain the defect probability value of each pixel point in the image to be detected, and combining to obtain the defect detection result image of the image to be detected.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 6.
9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210200679.3A CN114299066B (en) | 2022-03-03 | 2022-03-03 | Defect detection method and device based on salient feature pre-extraction and image segmentation |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210200679.3A CN114299066B (en) | 2022-03-03 | 2022-03-03 | Defect detection method and device based on salient feature pre-extraction and image segmentation |
Publications (2)
Publication Number | Publication Date |
---|---|
CN114299066A CN114299066A (en) | 2022-04-08 |
CN114299066B true CN114299066B (en) | 2022-05-31 |
Family
ID=80978622
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202210200679.3A Active CN114299066B (en) | 2022-03-03 | 2022-03-03 | Defect detection method and device based on salient feature pre-extraction and image segmentation |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN114299066B (en) |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114549997B (en) * | 2022-04-27 | 2022-07-29 | 清华大学 | X-ray image defect detection method and device based on regional feature extraction |
CN114926441A (en) * | 2022-05-26 | 2022-08-19 | 昆山缔微致精密电子有限公司 | Defect detection method and system for machining and molding injection molding part |
CN115100208B (en) * | 2022-08-26 | 2024-01-12 | 山东蓝海晶体科技有限公司 | Film surface defect evaluation method based on histogram and dynamic light source |
CN117132563B (en) * | 2023-08-24 | 2024-07-16 | 广东理工学院 | Glass defect detection method and device, electronic equipment and storage medium |
CN117036175B (en) * | 2023-10-08 | 2024-01-09 | 之江实验室 | Linear array image splicing method, device, medium and equipment |
CN117078677B (en) * | 2023-10-16 | 2024-01-30 | 江西天鑫冶金装备技术有限公司 | Defect detection method and system for starting sheet |
CN117541832B (en) * | 2024-01-04 | 2024-04-16 | 苏州镁伽科技有限公司 | Abnormality detection method, abnormality detection system, electronic device, and storage medium |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113538433A (en) * | 2021-09-17 | 2021-10-22 | 海门市创睿机械有限公司 | Mechanical casting defect detection method and system based on artificial intelligence |
CN113781402A (en) * | 2021-08-19 | 2021-12-10 | 中国电子产品可靠性与环境试验研究所((工业和信息化部电子第五研究所)(中国赛宝实验室)) | Method and device for detecting chip surface scratch defects and computer equipment |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110570396B (en) * | 2019-08-07 | 2022-03-15 | 华中科技大学 | Industrial product defect detection method based on deep learning |
CN110717896B (en) * | 2019-09-24 | 2023-05-09 | 东北大学 | Plate strip steel surface defect detection method based on significance tag information propagation model |
WO2021146935A1 (en) * | 2020-01-21 | 2021-07-29 | 京东方科技集团股份有限公司 | Image defect determining method and apparatus, and electronic device and storage medium |
CN112767369A (en) * | 2021-01-25 | 2021-05-07 | 佛山科学技术学院 | Defect identification and detection method and device for small hardware and computer readable storage medium |
CN113657383B (en) * | 2021-08-24 | 2024-05-24 | 凌云光技术股份有限公司 | Defect region detection method and device based on lightweight segmentation model |
CN113421263B (en) * | 2021-08-24 | 2021-11-30 | 深圳市信润富联数字科技有限公司 | Part defect detection method, device, medium and computer program product |
CN113888461A (en) * | 2021-08-26 | 2022-01-04 | 华能大理风力发电有限公司 | Method, system and equipment for detecting defects of hardware parts based on deep learning |
-
2022
- 2022-03-03 CN CN202210200679.3A patent/CN114299066B/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113781402A (en) * | 2021-08-19 | 2021-12-10 | 中国电子产品可靠性与环境试验研究所((工业和信息化部电子第五研究所)(中国赛宝实验室)) | Method and device for detecting chip surface scratch defects and computer equipment |
CN113538433A (en) * | 2021-09-17 | 2021-10-22 | 海门市创睿机械有限公司 | Mechanical casting defect detection method and system based on artificial intelligence |
Also Published As
Publication number | Publication date |
---|---|
CN114299066A (en) | 2022-04-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN114299066B (en) | Defect detection method and device based on salient feature pre-extraction and image segmentation | |
CN111553929B (en) | Mobile phone screen defect segmentation method, device and equipment based on converged network | |
US20210374940A1 (en) | Product defect detection method, device and system | |
CN110826416B (en) | Bathroom ceramic surface defect detection method and device based on deep learning | |
CN106875381B (en) | Mobile phone shell defect detection method based on deep learning | |
CN111814867A (en) | Defect detection model training method, defect detection method and related device | |
US20240233313A1 (en) | Model training method, image processing method, computing and processing device and non-transient computer-readable medium | |
CN113610822B (en) | Surface defect detection method based on multi-scale information fusion | |
CN111462120A (en) | Defect detection method, device, medium and equipment based on semantic segmentation model | |
CN111738344A (en) | Rapid target detection method based on multi-scale fusion | |
US11348349B2 (en) | Training data increment method, electronic apparatus and computer-readable medium | |
CN115965816B (en) | Glass defect classification and detection method and system based on deep learning | |
CN115880298A (en) | Glass surface defect detection method and system based on unsupervised pre-training | |
CN114663380A (en) | Aluminum product surface defect detection method, storage medium and computer system | |
CN112819748A (en) | Training method and device for strip steel surface defect recognition model | |
CN113298809A (en) | Composite material ultrasonic image defect detection method based on deep learning and superpixel segmentation | |
CN115797314A (en) | Part surface defect detection method, system, equipment and storage medium | |
CN116129242A (en) | Aluminum product surface defect identification method based on improved YOLOv4 | |
CN115239705A (en) | Method, device, equipment and storage medium for counting the number of endometrial plasma cells | |
CN112686896B (en) | Glass defect detection method based on frequency domain and space combination of segmentation network | |
CN114841992A (en) | Defect detection method based on cyclic generation countermeasure network and structural similarity | |
CN113569737A (en) | Notebook screen defect detection method and medium based on autonomous learning network model | |
CN113052798A (en) | Screen aging detection model training method and screen aging detection method | |
CN115630660B (en) | Barcode positioning method and device based on convolutional neural network | |
CN116824122A (en) | LED chip positioning method and LED chip positioning device based on deep learning |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |