CN114841977B - Defect detecting method based on Swin transducer structure combined with SSIM and GMSD - Google Patents

Defect detecting method based on Swin transducer structure combined with SSIM and GMSD Download PDF

Info

Publication number
CN114841977B
CN114841977B CN202210535974.4A CN202210535974A CN114841977B CN 114841977 B CN114841977 B CN 114841977B CN 202210535974 A CN202210535974 A CN 202210535974A CN 114841977 B CN114841977 B CN 114841977B
Authority
CN
China
Prior art keywords
picture
feature map
defect
swin
learning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210535974.4A
Other languages
Chinese (zh)
Other versions
CN114841977A (en
Inventor
江结林
朱加乐
郭浩然
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing University of Information Science and Technology
Original Assignee
Nanjing University of Information Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing University of Information Science and Technology filed Critical Nanjing University of Information Science and Technology
Priority to CN202210535974.4A priority Critical patent/CN114841977B/en
Publication of CN114841977A publication Critical patent/CN114841977A/en
Application granted granted Critical
Publication of CN114841977B publication Critical patent/CN114841977B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/761—Proximity, similarity or dissimilarity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30124—Fabrics; Textile; Paper
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a defect detection method based on a Swin transform structure combined with SSIM and GMSD, which comprises the steps of obtaining an industrial defect picture, inputting a pre-constructed characteristic extraction network based on the Swin transform to perform characteristic learning, and reconstructing defect-free area information of the learned characteristic picture to obtain a reconstructed picture; calculating the structural similarity SSIM of the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature map based on the structural similarity; calculating gradient amplitude similarity deviation GMSD between the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature map based on gradient amplitude similarity; the defect abnormal feature map based on the structural similarity is fused with the defect abnormal feature map based on the gradient amplitude similarity to obtain a final defect detection effect map.

Description

Defect detecting method based on Swin transducer structure combined with SSIM and GMSD
Technical Field
The invention relates to a defect detection method based on a Swin transform structure combined with SSIM and GMSD, and belongs to the technical field.
Background
Textile industry is an industry that accompanies the development of human civilization history, and is considered to be the most fundamental industry in human life. Fabric defect detection (also known as defect detection) is an essential step in quality control in textile industry production. If the surface of the fabric contains defects, the quality of the product is affected, the grading of the fabric is directly affected, and the price of the defective fabric is reduced. Thus, defect detection of fabrics appears to be critical in textile quality control.
The focus of fabric defect detection is to identify whether a given fabric image is abnormal relative to a set of normal samples and locate the corresponding abnormal region in the image. These two tasks have a great influence on the quality control in the textile industry. In the actual textile industry, fabric samples with defects rarely appear, and defects have a wide variety of shapes and textures, which are unpredictable. Therefore, the task of detecting defects in fabrics is difficult to process using supervised methods, and the method of generating formulas is preferred.
In recent years, due to the rapid development of deep learning, computer vision technology has been widely used in detecting defects in fabrics. Many researchers have directed their eyes towards convolutional neural networks, and classical full convolutional neural networks (FCNs), a nnet model, which is optimized via FCN networks and has the advantages of an automatic coding model, consisting of symmetrical codecs with jump connections, are proposed. In the encoder, a series of convolutional layers and successive downsampling layers are employed to extract the depth features of the large receive field. And then, the decoder upsamples the extracted depth features to the input resolution to perform pixel-level semantic prediction, and performs jump connection fusion on the high-resolution features of different scales output by the encoder so as to reduce the spatial information loss caused by downsampling. With such an elegant U-shaped structural design, a number of fault detection algorithms, such as MSCDAE, CASAE, etc., have been developed, and these U-shaped convolutional neural network models exhibit good performance in a variety of industrial detection applications.
At present, the reconstruction type defect detection algorithm based on the U-shaped convolutional neural network is in accordance with an unsupervised method, and only the distribution of normal data is modeled during training. An anomaly score for the fabric is calculated for the difference between the input image and the model reconstructed image at the time of testing. The whole detection process assumes that the model is insensitive to reconstruction of an abnormal region, but in practical application, the generalization capability of a convolution network is strong, and the reconstruction effect on the abnormality is very effective, so that the generated network has higher omission rate on defect detection.
Defects existing in the current defect detection mainly comprise: (1) the detection omission rate of the cloth defect detection is too high. (2) inaccurate defect localization.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provides a defect detection method based on a Swin transform structure combined with SSIM and GMSD, which can not only effectively improve the accuracy of defect detection, but also improve the accuracy of defect positioning and reduce the false detection rate.
In order to achieve the above purpose, the invention is realized by adopting the following technical scheme:
in a first aspect, the present invention provides a defect detection method based on a Swin transducer structure in combination with SSIM and GMSD, comprising:
acquiring an industrial defect picture, inputting a pre-constructed Swin transform-based feature extraction network to perform feature learning, and reconstructing defect-free region information of the learned feature picture to obtain a reconstructed picture;
calculating the structural similarity SSIM of the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature map based on the structural similarity;
calculating gradient amplitude similarity deviation GMSD between the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature map based on gradient amplitude similarity;
and fusing the defect abnormal characteristic map based on the structural similarity with the defect abnormal characteristic map based on the gradient amplitude similarity to obtain a final defect detection effect map.
Further, the step of obtaining the industrial defect picture, inputting a pre-constructed Swin Transformer-based feature extraction network to perform feature learning, and reconstructing the non-defect area information of the learned feature to obtain a reconstructed picture, including:
acquiring an industrial defect picture, and inputting a pre-constructed Swin transform-based feature extraction network, wherein the Swin transform-based feature extraction network comprises a Patch part module, a Swin Transformer Block module, a Patch measuring module and a PixelSheffle module; and the size of the industrial defect picture is H multiplied by W multiplied by 3;
the input industrial defect picture enters a Patch part module to convert the space information into channel information, so that the input industrial defect picture meets the input format of the Swin Transformer Block module;
inputting the converted industrial defect picture into a Swin Transformer Block module for feature learning to obtain a learned feature map;
and respectively inputting the learned feature images into a Patch measuring module or a PixelSheffle module to perform downsampling or upsampling operation to obtain a reconstructed image, wherein the reconstructed image only comprises a normal image area, and the information of the defect area is not reconstructed.
Further, the input industrial defect picture enters a Patch part module to convert the space information into channel information, which comprises:
carrying out convolution operation of 1×1 on an input industrial defect picture to obtain a first characteristic picture with the size of H×W×C, and carrying out 4 times downsampling on the obtained first characteristic picture to obtain a second characteristic picture with the size of H×W×C
Figure BDA0003648249370000031
Wherein->
Figure BDA0003648249370000032
Representing the number of patches, C represents the information dimension of a patch.
Further, inputting the transformed industrial defect picture to the Swin Transformer Block module for feature learning to obtain a learned feature map, including:
sending the second feature map to a Swin Transformer Block module for learning, and performing layer normalization operation on the input industrial defect picture;
performing a multi-headed self-attention operation, learning a relational representation between local patches;
adding the patch representation after the multi-head self-attention operation is executed with the original input to obtain a new patch representation;
performing LN operation on the obtained new patch, then performing multi-layer sensor operation, and further fitting the patch;
adding the fitted patch to the obtained new patch to form a final patch representation obtained through multi-head self-attention stage learning;
and sending the obtained final patch representation to multi-head self-attention stage learning with a moving window to obtain a patch representation containing both local relation information and global relation information, thereby obtaining a third characteristic diagram after learning.
Further, the inputting the learned feature map into the Patch berging module or the PixelShuffle module to perform downsampling or upsampling operation, respectively, to obtain a reconstructed picture, includes:
downsampling the third feature map learned by the Swin Transformer Block module by 2 times to obtain a fourth feature map with the size of
Figure BDA0003648249370000041
The fourth feature map is subjected to Swin Transformer Block module learning and then downsampled for 2 times to obtain a fifth feature map with the size of
Figure BDA0003648249370000042
The fifth feature map is subjected to Swin Transformer Block module learning and then downsampled for 2 times to obtain a sixth feature map with the size of
Figure BDA0003648249370000043
The sixth feature map is subjected to Swin Transformer Block module learning and then is up-sampled for 2 times to obtain a seventh feature map with the size of
Figure BDA0003648249370000051
Adding the seventh feature map and the fifth feature map, learning by a Swin Transformer Block module, and then up-sampling by 2 times to obtain an eighth feature map with the size of
Figure BDA0003648249370000052
Adding the eighth feature map and the fourth feature map, learning by a Swin Transformer Block module, and then up-sampling by 2 times to obtain a ninth feature map with the size of
Figure BDA0003648249370000053
And adding the ninth feature map and the third feature map, and up-sampling by 4 times after learning by a Swin Transformer Block module to obtain a reconstructed picture with the size of H multiplied by W multiplied by 3.
Further, calculating the structural similarity SSIM of the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature picture based on the structural similarity, wherein the formula is as follows:
Figure BDA0003648249370000054
wherein X is the original input picture, Y is the reconstructed picture, u X 、u Y Representing the mean, delta, of images X and Y XY Representing the covariance of images X and Y, delta X 2 、δ Y 2 Representing variances representing images X and Y, c 1 、c 2 Is constant.
Further, calculating a gradient amplitude similarity deviation GMSD between the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature map based on gradient amplitude similarity, wherein the formula is as follows:
Figure BDA0003648249370000055
wherein g (X) and g (Y) represent gradient amplitude graphs of X and Y, and the calculation formula is as follows:
Figure BDA0003648249370000056
/>
Figure BDA0003648249370000057
wherein h is x And h y Is a 3 x 3 Prewitt filter along the x and y dimensions and is a convolution operation.
Further, fusing the defect abnormal feature map based on the structural similarity with the defect abnormal feature map based on the gradient amplitude similarity to obtain a final defect detection effect map, including:
will M S And M g Removing M by Gaussian smoothing S And M g Abnormal value in (1) and then smoothing the smoothed M S And M g And adding and averaging to obtain a final detection result M.
In a second aspect, the present invention provides a defect detection device based on a swinTransformer structure in combination with SSIM and GMSD, comprising:
the characteristic learning and reconstructing unit is used for acquiring an industrial defect picture, inputting a pre-constructed Swin Transformer-based characteristic extraction network to perform characteristic learning, and reconstructing the non-defect area information of the learned characteristic picture to obtain a reconstructed picture;
the structural similarity calculating unit is used for calculating the structural similarity SSIM of the industrial defect picture and the reconstructed picture to obtain a defect abnormal characteristic diagram based on the structural similarity;
the gradient amplitude similarity deviation calculation unit is used for calculating the gradient amplitude similarity deviation GMSD between the industrial defect picture and the reconstructed picture to obtain a defect abnormal characteristic diagram based on gradient amplitude similarity;
and the feature fusion unit is used for fusing the defect abnormal feature map based on the structural similarity with the defect abnormal feature map based on the gradient amplitude similarity to obtain a final defect detection effect map.
In a third aspect, the present invention provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of any of the methods described in the preceding claims.
Compared with the prior art, the invention has the beneficial effects that:
the invention provides a defect detection method based on a SwinTransformer structure and combined with SSIM and GMSD, which is characterized in that a novel cloth defect detection method based on deep learning is established through a characteristic extraction network based on the SwinTransformer structure, the aim of intelligent detection of industrial materials is fulfilled, compared with the existing defect detection algorithm, the characteristic extraction network based on the SwinTransformer structure is used for extracting and reconstructing the image by utilizing the SwinTransformer structure, global information in the image can be fully utilized, the reconstruction of the image is facilitated, the difference value between an input image and a reconstructed image is calculated in a model by utilizing Structural Similarity (SSIM) and gradient amplitude similarity deviation (GMSD), the integrity and global gradient information of the image can be fully considered, and the final defect detection result is obtained by fusing the defect abnormal characteristic image calculated by the SSIM and the defect positioning accuracy of the algorithm.
Drawings
FIG. 1 is a flow chart of a defect detection method based on a Swin transducer structure combined with SSIM and GMSD according to an embodiment of the present invention;
FIG. 2 is a flowchart of a defect detection algorithm provided by an embodiment of the present invention;
FIG. 3 is a diagram of a Swin TransformerNet architecture provided by an embodiment of the invention;
FIG. 4 is a schematic illustration of an industrial defect picture provided by an embodiment of the present invention;
FIG. 5 is a schematic view of a real label of an industrial defect picture provided by an embodiment of the present invention;
FIG. 6 is a graph of the resulting defect detection effect provided by embodiments of the present invention.
Detailed Description
The invention is further described below with reference to the accompanying drawings. The following examples are only for more clearly illustrating the technical aspects of the present invention, and are not intended to limit the scope of the present invention.
Example 1
This embodiment describes a defect detection method based on a Swin transducer structure in combination with SSIM and GMSD, comprising:
acquiring an industrial defect picture, inputting a pre-constructed Swin transform-based feature extraction network to perform feature learning, and reconstructing defect-free region information of the learned feature picture to obtain a reconstructed picture;
calculating the structural similarity SSIM of the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature map based on the structural similarity;
calculating gradient amplitude similarity deviation GMSD between the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature map based on gradient amplitude similarity;
and fusing the defect abnormal characteristic map based on the structural similarity with the defect abnormal characteristic map based on the gradient amplitude similarity to obtain a final defect detection effect map.
The defect detection method based on the Swin transducer structure combined with the SSIM and the GMSD provided by the embodiment specifically comprises the following steps:
step 10: and inputting the input original image (X) into the model to learn the characteristics, so as to realize the reconstruction of the information of the non-defect area, and finally obtaining the network reconstruction picture (Y). Specific network model as in fig. 3, the model is explained in detail below:
Figure BDA0003648249370000081
/>
Figure BDA0003648249370000091
/>
Figure BDA0003648249370000101
step 20: calculating the Structural Similarity (SSIM) between the input original picture (X) and the reconstructed picture (Y) obtained in the step 10 to obtain a defect anomaly characteristic map (M) based on the structural similarity S ) The SSIM calculation formula is as follows:
Figure BDA0003648249370000102
wherein X is the original input picture, Y is the reconstructed picture, u X 、u Y Representing the mean, delta, of images X and Y XY Representing the covariance of images X and Y, delta X 2 、δ Y 2 Representation ofRepresenting the variance of images X and Y, c 1 、c 2 Is constant.
Step 30: calculating gradient amplitude similarity deviation (GMSD) between an input original picture and a reconstructed picture to obtain a defect abnormal feature map (M) based on gradient amplitude similarity g ) The GMS calculation formula is as follows:
Figure BDA0003648249370000103
wherein g (X) and g (Y) represent gradient amplitude graphs of X and Y, and the calculation formula is as follows:
Figure BDA0003648249370000104
Figure BDA0003648249370000105
wherein h is x And h y Is a 3 x 3 Prewitt filter along the x and y dimensions and is a convolution operation.
Step 40: m obtained by step 20 and step 30 S And M g And fusing to obtain a final defect detection effect diagram. The specific operation is as follows: will M S And M g Removing M by Gaussian smoothing S And M g Is an outlier in (a). Then M after smoothing S And M g And adding and averaging to obtain a final detection result M.
Example 2
The embodiment provides a defect detection device based on a SwinTransformer structure combined with SSIM and GMSD, which comprises:
the characteristic learning and reconstructing unit is used for acquiring an industrial defect picture, inputting a pre-constructed Swin Transformer-based characteristic extraction network to perform characteristic learning, and reconstructing the non-defect area information of the learned characteristic picture to obtain a reconstructed picture;
the structural similarity calculating unit is used for calculating the structural similarity SSIM of the industrial defect picture and the reconstructed picture to obtain a defect abnormal characteristic diagram based on the structural similarity;
the gradient amplitude similarity deviation calculation unit is used for calculating the gradient amplitude similarity deviation GMSD between the industrial defect picture and the reconstructed picture to obtain a defect abnormal characteristic diagram based on gradient amplitude similarity;
and the feature fusion unit is used for fusing the defect abnormal feature map based on the structural similarity with the defect abnormal feature map based on the gradient amplitude similarity to obtain a final defect detection effect map.
Example 3
The present embodiment provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of any of the methods described in the preceding claims.
The foregoing is merely a preferred embodiment of the present invention, and it should be noted that modifications and variations could be made by those skilled in the art without departing from the technical principles of the present invention, and such modifications and variations should also be regarded as being within the scope of the invention.

Claims (4)

1. A defect detection method based on a Swin transducer structure combined with SSIM and GMSD, comprising:
acquiring an industrial defect picture, inputting a pre-constructed Swin transform-based feature extraction network to perform feature learning, and reconstructing defect-free region information of the learned feature picture to obtain a reconstructed picture;
calculating the structural similarity SSIM of the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature map based on the structural similarity;
calculating gradient amplitude similarity deviation GMSD between the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature map based on gradient amplitude similarity;
fusing the defect abnormal feature map based on the structural similarity with the defect abnormal feature map based on the gradient amplitude similarity to obtain a final defect detection effect map;
the method for obtaining the industrial defect picture, inputting a pre-constructed Swin transform-based feature extraction network for feature learning, reconstructing the learned feature with no defect region information, and obtaining a reconstructed picture, comprises the following steps:
acquiring an industrial defect picture, and inputting a pre-constructed Swin transform-based feature extraction network, wherein the Swin transform-based feature extraction network comprises a Patch part module, a Swin Transformer Block module, a Patch measuring module and a PixelSheffle module; and the size of the industrial defect picture is H multiplied by W multiplied by 3;
the input industrial defect picture enters a Patch part module to convert the space information into channel information, so that the input industrial defect picture meets the input format of the Swin Transformer Block module;
inputting the converted industrial defect picture into a Swin Transformer Block module for feature learning to obtain a learned feature map;
respectively inputting the learned feature images into a Patch measuring module and a PixelSheffle module to perform downsampling and upsampling operations to obtain a reconstructed image, wherein the reconstructed image only comprises a normal image area, and the information of the defect area is not reconstructed;
the input industrial defect picture enters a Patch part module to convert space information into channel information, and the method comprises the following steps:
carrying out convolution operation of 1×1 on an input industrial defect picture to obtain a first characteristic picture with the size of H×W×C, and carrying out 4 times downsampling on the obtained first characteristic picture to obtain a second characteristic picture with the size of H×W×C
Figure FDA0004127916680000021
Wherein->
Figure FDA0004127916680000022
Representing the number of the patches, C representing the information dimension of the patches;
inputting the converted industrial defect picture into a Swin Transformer Block module for feature learning to obtain a learned feature map, wherein the feature map comprises the following components:
sending the second feature map to a Swin Transformer Block module for learning, and performing layer normalization operation on the input industrial defect picture;
performing a multi-headed self-attention operation, learning a relational representation between local patches;
adding the patch representation after the multi-head self-attention operation is executed with the original input to obtain a new patch representation;
performing LN operation on the obtained new patch, then performing multi-layer sensor operation, and further fitting the patch;
adding the fitted patch to the obtained new patch to form a final patch representation obtained through multi-head self-attention stage learning;
sending the obtained final patch representation to multi-head self-attention stage learning with a moving window to obtain a patch representation containing local relation information and global relation information, thereby obtaining a third characteristic diagram after learning;
the step of inputting the learned feature map into a Patch Merging module and a PixelSheffe module for downsampling and upsampling respectively to obtain a reconstructed picture comprises the following steps:
downsampling the third feature map learned by the Swin Transformer Block module by 2 times to obtain a fourth feature map with the size of
Figure FDA0004127916680000031
The fourth feature map is subjected to Swin Transformer Block module learning and then downsampled for 2 times to obtain a fifth feature map with the size of
Figure FDA0004127916680000032
The fifth feature map is subjected to Swin Transformer Block module learning and then downsampled for 2 times to obtain a sixth feature map with the size of
Figure FDA0004127916680000033
The sixth feature map is subjected to Swin Transformer Block module learning and then is up-sampled for 2 times to obtain a seventh feature map with the size of
Figure FDA0004127916680000034
Adding the seventh feature map and the fifth feature map, learning by a Swin Transformer Block module, and then up-sampling by 2 times to obtain an eighth feature map with the size of
Figure FDA0004127916680000035
Adding the eighth feature map and the fourth feature map, learning by a Swin Transformer Block module, and then up-sampling by 2 times to obtain a ninth feature map with the size of
Figure FDA0004127916680000036
Adding the ninth feature map and the third feature map, and performing up-sampling for 4 times after learning by a Swin Transformer Block module to obtain a reconstructed picture with the size of H multiplied by W multiplied by 3;
calculating the structural similarity SSIM of the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature map M based on the structural similarity S The formula is as follows:
Figure FDA0004127916680000037
wherein X is the original input picture, Y is the reconstructed picture, u X 、u Y Representing the mean, delta, of images X and Y XY Representing the covariance of images X and Y, delta X 2 、δ Y 2 Representing variances representing images X and Y, c 1 、c 2 Is a constant;
calculating gradient amplitude similarity deviation GMSD between the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature map M based on gradient amplitude similarity g Formulas such asThe following steps:
Figure FDA0004127916680000041
wherein g (X) and g (Y) represent gradient amplitude graphs of X and Y, and the calculation formula is as follows:
Figure FDA0004127916680000042
Figure FDA0004127916680000043
wherein h is x And h y Is a 3 x 3 Prewitt filter along the x and y dimensions and is a convolution operation.
2. A method of defect detection based on a Swin transducer structure in combination with SSIM and GMSD as claimed in claim 1, wherein: fusing the defect abnormal feature map based on the structural similarity with the defect abnormal feature map based on the gradient amplitude similarity to obtain a final defect detection effect map, wherein the method comprises the following steps of:
will M S And M g Removing M by Gaussian smoothing S And M g Abnormal value in (1) and then smoothing the smoothed M S And M g And adding and averaging to obtain a final detection result M.
3. A defect detection device based on a Swin transducer structure combined with SSIM and GMSD, comprising:
the characteristic learning and reconstructing unit is used for acquiring an industrial defect picture, inputting a pre-constructed Swin Transformer-based characteristic extraction network to perform characteristic learning, and reconstructing the non-defect area information of the learned characteristic picture to obtain a reconstructed picture;
the structural similarity calculating unit is used for calculating the structural similarity SSIM of the industrial defect picture and the reconstructed picture to obtain a defect abnormal characteristic diagram based on the structural similarity;
the gradient amplitude similarity deviation calculation unit is used for calculating the gradient amplitude similarity deviation GMSD between the industrial defect picture and the reconstructed picture to obtain a defect abnormal characteristic diagram based on gradient amplitude similarity;
the feature fusion unit is used for fusing the defect abnormal feature map based on the structural similarity with the defect abnormal feature map based on the gradient amplitude similarity to obtain a final defect detection effect map;
wherein: the method for obtaining the industrial defect picture, inputting a pre-constructed Swin transform-based feature extraction network for feature learning, reconstructing the learned feature with no defect region information, and obtaining a reconstructed picture, comprises the following steps:
acquiring an industrial defect picture, and inputting a pre-constructed Swin transform-based feature extraction network, wherein the Swin transform-based feature extraction network comprises a Patch part module, a Swin Transformer Block module, a Patch measuring module and a PixelSheffle module; and the size of the industrial defect picture is H multiplied by W multiplied by 3;
the input industrial defect picture enters a Patch part module to convert the space information into channel information, so that the input industrial defect picture meets the input format of the Swin Transformer Block module;
inputting the converted industrial defect picture into a Swin Transformer Block module for feature learning to obtain a learned feature map;
respectively inputting the learned feature images into a Patch measuring module and a PixelSheffle module to perform downsampling and upsampling operations to obtain a reconstructed image, wherein the reconstructed image only comprises a normal image area, and the information of the defect area is not reconstructed;
the input industrial defect picture enters a Patch part module to convert space information into channel information, and the method comprises the following steps:
the input industrial defect picture is subjected to 1X 1 convolution operation to obtain a first characteristic picture with the size of H X W X C, and the obtained first characteristic picture is subjected to 4 times lower than the first characteristic pictureSampling to obtain a second characteristic diagram with the size of
Figure FDA0004127916680000051
Wherein->
Figure FDA0004127916680000052
Representing the number of the patches, C representing the information dimension of the patches;
inputting the converted industrial defect picture into a Swin Transformer Block module for feature learning to obtain a learned feature map, wherein the feature map comprises the following components:
sending the second feature map to a Swin Transformer Block module for learning, and performing layer normalization operation on the input industrial defect picture;
performing a multi-headed self-attention operation, learning a relational representation between local patches;
adding the patch representation after the multi-head self-attention operation is executed with the original input to obtain a new patch representation;
performing LN operation on the obtained new patch, then performing multi-layer sensor operation, and further fitting the patch;
adding the fitted patch to the obtained new patch to form a final patch representation obtained through multi-head self-attention stage learning;
sending the obtained final patch representation to multi-head self-attention stage learning with a moving window to obtain a patch representation containing local relation information and global relation information, thereby obtaining a third characteristic diagram after learning;
the step of inputting the learned feature map into a Patch Merging module and a PixelSheffe module for downsampling and upsampling respectively to obtain a reconstructed picture comprises the following steps:
downsampling the third feature map learned by the Swin Transformer Block module by 2 times to obtain a fourth feature map with the size of
Figure FDA0004127916680000061
Subjecting the fourth feature map to Swin transformationDownsampling 2 times after r Block module learning to obtain a fifth feature map with the size of
Figure FDA0004127916680000062
The fifth feature map is subjected to Swin Transformer Block module learning and then downsampled for 2 times to obtain a sixth feature map with the size of
Figure FDA0004127916680000063
The sixth feature map is subjected to Swin Transformer Block module learning and then is up-sampled for 2 times to obtain a seventh feature map with the size of
Figure FDA0004127916680000071
Adding the seventh feature map and the fifth feature map, learning by a Swin Transformer Block module, and then up-sampling by 2 times to obtain an eighth feature map with the size of
Figure FDA0004127916680000072
Adding the eighth feature map and the fourth feature map, learning by a Swin Transformer Block module, and then up-sampling by 2 times to obtain a ninth feature map with the size of
Figure FDA0004127916680000073
Adding the ninth feature map and the third feature map, and performing up-sampling for 4 times after learning by a Swin Transformer Block module to obtain a reconstructed picture with the size of H multiplied by W multiplied by 3;
calculating the structural similarity SSIM of the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature map M based on the structural similarity S The formula is as follows:
Figure FDA0004127916680000074
wherein X is the original input picture, Y is the reconstructed picture, u X 、u Y Representing the mean, delta, of images X and Y XY Representing the covariance of images X and Y, delta X 2 、δ Y 2 Representing variances representing images X and Y, c 1 、c 2 Is a constant;
calculating gradient amplitude similarity deviation GMSD between the industrial defect picture and the reconstructed picture to obtain a defect abnormal feature map M based on gradient amplitude similarity g The formula is as follows:
Figure FDA0004127916680000075
wherein g (X) and g (Y) represent gradient amplitude graphs of X and Y, and the calculation formula is as follows:
Figure FDA0004127916680000076
Figure FDA0004127916680000077
wherein h is x And h y Is a 3 x 3 Prewitt filter along the x and y dimensions and is a convolution operation.
4. A computer-readable storage medium having stored thereon a computer program, characterized by: the program being operative to perform the steps of the method as claimed in any one of claims 1 to 2 when executed by a processor.
CN202210535974.4A 2022-05-17 2022-05-17 Defect detecting method based on Swin transducer structure combined with SSIM and GMSD Active CN114841977B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210535974.4A CN114841977B (en) 2022-05-17 2022-05-17 Defect detecting method based on Swin transducer structure combined with SSIM and GMSD

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210535974.4A CN114841977B (en) 2022-05-17 2022-05-17 Defect detecting method based on Swin transducer structure combined with SSIM and GMSD

Publications (2)

Publication Number Publication Date
CN114841977A CN114841977A (en) 2022-08-02
CN114841977B true CN114841977B (en) 2023-04-25

Family

ID=82568804

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210535974.4A Active CN114841977B (en) 2022-05-17 2022-05-17 Defect detecting method based on Swin transducer structure combined with SSIM and GMSD

Country Status (1)

Country Link
CN (1) CN114841977B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115330898B (en) * 2022-08-24 2023-06-06 晋城市大锐金马工程设计咨询有限公司 Magazine advertisement embedding method based on improved Swin Transformer

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111932501A (en) * 2020-07-13 2020-11-13 太仓中科信息技术研究院 Seal ring surface defect detection method based on semantic segmentation
CN112116543A (en) * 2020-09-25 2020-12-22 中国科学院自动化研究所 Image restoration method, system and device based on detection type generation framework
CN112802074A (en) * 2021-01-06 2021-05-14 江南大学 Textile flaw detection method based on illumination correction and visual saliency characteristics
CN113902695A (en) * 2021-09-29 2022-01-07 西安工程大学 Detection method for yarn-dyed fabric cut piece defect area
CN114387207A (en) * 2021-12-02 2022-04-22 浙江大学 Tire flaw detection method and model based on self-attention mechanism and dual-field self-adaptation

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8620092B2 (en) * 2010-03-04 2013-12-31 Hewlett-Packard Development Company, L.P. Determining similarity of two images
US10970856B2 (en) * 2018-12-27 2021-04-06 Baidu Usa Llc Joint learning of geometry and motion with three-dimensional holistic understanding
AU2019201467A1 (en) * 2019-03-04 2020-09-24 Canon Kabushiki Kaisha System and method of reducing effects of turbulence in an image
US20200380290A1 (en) * 2019-05-31 2020-12-03 Apple Inc. Machine learning-based prediction of precise perceptual video quality
CN113989228A (en) * 2021-10-27 2022-01-28 西安工程大学 Method for detecting defect area of color texture fabric based on self-attention
CN114022442B (en) * 2021-11-03 2022-11-29 武汉智目智能技术合伙企业(有限合伙) Unsupervised learning-based fabric defect detection algorithm
CN114037684B (en) * 2021-11-08 2024-06-14 南京信息工程大学 Defect detection method based on yolov and attention mechanism model
CN114140353B (en) * 2021-11-25 2023-04-07 苏州大学 Swin-Transformer image denoising method and system based on channel attention

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111932501A (en) * 2020-07-13 2020-11-13 太仓中科信息技术研究院 Seal ring surface defect detection method based on semantic segmentation
CN112116543A (en) * 2020-09-25 2020-12-22 中国科学院自动化研究所 Image restoration method, system and device based on detection type generation framework
CN112802074A (en) * 2021-01-06 2021-05-14 江南大学 Textile flaw detection method based on illumination correction and visual saliency characteristics
CN113902695A (en) * 2021-09-29 2022-01-07 西安工程大学 Detection method for yarn-dyed fabric cut piece defect area
CN114387207A (en) * 2021-12-02 2022-04-22 浙江大学 Tire flaw detection method and model based on self-attention mechanism and dual-field self-adaptation

Also Published As

Publication number Publication date
CN114841977A (en) 2022-08-02

Similar Documents

Publication Publication Date Title
WO2023070911A1 (en) Self-attention-based method for detecting defective area of color-textured fabric
CN111626994A (en) Equipment fault defect diagnosis method based on improved U-Net neural network
CN114782694B (en) Unsupervised anomaly detection method, system, device and storage medium
Shao et al. Uncertainty guided multi-scale attention network for raindrop removal from a single image
CN114742799B (en) Industrial scene unknown type defect segmentation method based on self-supervision heterogeneous network
CN114841977B (en) Defect detecting method based on Swin transducer structure combined with SSIM and GMSD
CN116051549B (en) Method, system, medium and equipment for dividing defects of solar cell
CN114170144A (en) Power transmission line pin defect detection method, equipment and medium
CN114170475A (en) Model training method and device for image classification and storage medium
CN115239642A (en) Detection method, detection device and equipment for hardware defects in power transmission line
CN115546046A (en) Single image defogging method fusing frequency and content characteristics
CN112070712A (en) Printing defect detection method based on self-encoder network
Li et al. Fabric defect segmentation system based on a lightweight GAN for industrial Internet of Things
Shit et al. An encoder‐decoder based CNN architecture using end to end dehaze and detection network for proper image visualization and detection
CN117253034A (en) Image semantic segmentation method and system based on differentiated context
CN114926679B (en) Image classification system and method for performing countermeasure defense
CN116721057A (en) Defect detection method and device based on transducer
CN115512108A (en) Semi-supervised OCT image retina segmentation method based on uncertainty
Guo et al. A water-level measurement method using sparse representation
CN115761606A (en) Box electric energy meter identification method and device based on image processing
CN115619707A (en) Method and system for detecting surface defects of sealing element of automobile door
CN112529881B (en) Power control cabinet cable anomaly identification method and device
CN114758133A (en) Image flaw segmentation method based on super-pixel active learning and semi-supervised learning strategies
CN111985625B (en) Infrared-visible light fused deep neural network and modeling method thereof
JP2023038060A (en) Label generation method, model generation method, label generation device, label generation program, model generation device, and model generation program

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB03 Change of inventor or designer information

Inventor after: Jiang Jielin

Inventor after: Zhu Jiale

Inventor after: Guo Haoran

Inventor before: Zhu Jiale

Inventor before: Jiang Jielin

Inventor before: Guo Haoran

CB03 Change of inventor or designer information
GR01 Patent grant
GR01 Patent grant