CN114820379B - Image rain-like layer removing method for generating countermeasure network based on attention dual residual error - Google Patents
Image rain-like layer removing method for generating countermeasure network based on attention dual residual error Download PDFInfo
- Publication number
- CN114820379B CN114820379B CN202210518394.4A CN202210518394A CN114820379B CN 114820379 B CN114820379 B CN 114820379B CN 202210518394 A CN202210518394 A CN 202210518394A CN 114820379 B CN114820379 B CN 114820379B
- Authority
- CN
- China
- Prior art keywords
- image
- rain
- layer
- attention
- generator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 230000009977 dual effect Effects 0.000 title claims abstract description 29
- 230000007547 defect Effects 0.000 claims abstract description 55
- 229910000831 Steel Inorganic materials 0.000 claims abstract description 44
- 239000010959 steel Substances 0.000 claims abstract description 44
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims abstract description 41
- 238000004519 manufacturing process Methods 0.000 claims abstract description 17
- 238000010276 construction Methods 0.000 claims abstract description 8
- 238000012549 training Methods 0.000 claims description 25
- 238000012360 testing method Methods 0.000 claims description 15
- 230000004927 fusion Effects 0.000 claims description 6
- 239000008186 active pharmaceutical agent Substances 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 5
- 238000009826 distribution Methods 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 3
- 239000000203 mixture Substances 0.000 claims description 3
- 238000005096 rolling process Methods 0.000 claims description 3
- 238000004458 analytical method Methods 0.000 claims description 2
- 238000004364 calculation method Methods 0.000 claims 1
- 239000002245 particle Substances 0.000 claims 1
- 238000004088 simulation Methods 0.000 claims 1
- 238000001514 detection method Methods 0.000 abstract description 7
- 230000008485 antagonism Effects 0.000 abstract 1
- 238000005457 optimization Methods 0.000 abstract 1
- 230000006870 function Effects 0.000 description 6
- 238000007689 inspection Methods 0.000 description 4
- 239000000498 cooling water Substances 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000000737 periodic effect Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000011158 quantitative evaluation Methods 0.000 description 2
- 102100022778 POC1 centriolar protein homolog A Human genes 0.000 description 1
- 101710125073 POC1 centriolar protein homolog A Proteins 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000001816 cooling Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 239000007921 spray Substances 0.000 description 1
- 238000011179 visual inspection Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30136—Metal
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Processing (AREA)
Abstract
The invention provides an image rain-like layer removing method for generating an antagonism network based on an attention dual residual error, which comprises the following steps: improving an image database construction method for removing a rain-like layer on the surface of hot rolled strip steel, and manufacturing a paired image database containing dispersed water drops, splash water lines and fine white water drops; inputting the clean background original image and the corresponding rain-like layer image into the attention dual residual error pair to generate an countermeasure network model, and introducing an attention scheme between a generator and a discriminator to form a self-optimization closed loop so as to mine priori knowledge of a generalized rain-like layer; and positioning and removing the rain-like pseudo defects by using the trained generator model. The invention can remove the mixed rain-like false defect clusters on the surface of the hot rolled strip steel on the premise of keeping the edge and texture details, the obtained result is closer to a real industrial image, and the false detection rate of an AVI instrument can be effectively reduced.
Description
Technical Field
The invention belongs to the technical field of digital image processing, and particularly relates to an image rain-like layer removing method for generating an countermeasure network based on attention dual residual errors.
Background
Steel is one of the basic materials of manufacturing enterprises, and the quality of the steel seriously affects the production of a plurality of subsequent industrial chains. Automatic Visual Inspection (AVI) is of great value in ensuring the quality of steel products, which is always placed at the back end of the spray cooling process. Under the influence of continuous industrial cooling water, a large number of scattered water drops, splashed water waves and tiny white water drops (essentially pseudo defects) randomly appear on the surface of the steel body and are mixed with real defects, so that false alarms are inevitably generated by an AVI instrument. More troublesome, part of the real defects are covered by the pseudo defects, so that the detection accuracy and efficiency of the AVI instrument are drastically reduced.
Many studies have attempted to detect and classify various defects directly from the raw strip images acquired under the above-described severe environments, but have had certain limitations in practical applications. Defects in hot rolled strip can be generally divided into two categories: one type is periodic defects and the other type is occasional defects. The frequency of sporadic defects is low, so that it is difficult to accumulate enough samples for neural network learning, but the sporadic defects are not negligible in steel quality detection. Therefore, algorithms based on statistical learning are often used in practical industrial lines, but such algorithms suffer from rain-like artifacts.
Underrain and past rain phenomena are also a major challenge in rain-like layer removal. Under the condition of no rain, the false alarm frequency of the AVI instrument is increased due to insufficient removal of the rain-like false defects. In contrast, rain in the past has erroneously removed some real defects, greatly affecting the accuracy of defect detection. The primary task to solve this problem is to construct an image restoration method with strong robustness to prevent the underraining and past rain problems.
It is also a difficulty how to build an image database for rain-like layer removal in the field of automatic detection of hot rolled strip surfaces. The conditions of cooling water dispersion, mechanical vibration, high temperature and the like frequently occur in the field image acquisition of the surface of the hot rolled strip steel, and the severe working conditions have strict requirements on data acquisition, so that the acquisition cost is high. Furthermore, since the strip moves at a fairly rapid rate on the production line, it is difficult to ensure that a pair of images with the same background are provided for training. Unfortunately, the common data set of hot rolled strip is very rare, which greatly limits the development of hot rolled strip rain-like removal algorithms.
In summary, the existing rain streak and rain line removing method cannot prevent the problem of underrain caused by high-density areas due to uneven distribution of cooling water in the image. Therefore, these algorithms do not meet the needs of the actual industry,
Disclosure of Invention
The invention aims to overcome the defects of the background technology, and provides a hot rolled steel strip surface image rain-like layer removing method based on attention dual residual error generation countermeasure network, which can remove mixed rain-like pseudo defect clusters on the hot rolled steel strip surface while maintaining edge and texture details.
The invention solves the technical problems by adopting the technical scheme that the method for removing the image rain-like layer based on the attention dual residual error generates an countermeasure network comprises a method for constructing pseudo-rain-like defects, a hot rolled strip steel surface rain-like layer image database and the attention dual residual error generates the countermeasure network; the method for constructing the rain-like pseudo defect is used for restoring the shape and the distribution position of the mixed rain-like pseudo defect cluster in a real steel mill on a clean original strip steel image and manufacturing a hot-rolled strip steel surface rain-like layer image database; the hot rolled strip steel surface rain-like layer image database is used for training and testing analysis of the attention dual residual error generation countermeasure network model; the attention dual residual error generation countermeasure network model is used for removing rain-like pseudo defects on the strip steel image on the premise of retaining the real defect edges and texture details. The method is implemented according to the following steps:
s10: aiming at three types of rain pseudo defects of scattered water drops, splashing water lines and tiny white water drops, a corresponding construction method is designed, and a rain-like layer image database on the surface of the hot-rolled strip steel is manufactured;
s20: building an attention dual residual error to generate an countermeasure network model, wherein the countermeasure network model consists of a generator G, a director M and a discriminator D;
S30: and (3) dividing the rainy layer image database manufactured in the step (S10) into a training set and a testing set.
S40: inputting the training set obtained in the step S30 into the attention dual residual error built in the step S20 in pairs to generate an countermeasure network model, introducing an attention scheme in the training process to mine priori knowledge of a generalized rain-like layer, repeatedly iterating and updating weights and losses of a generator and a discriminator, and obtaining a plurality of iteration version generation models.
S50: inputting the image of the testing set containing the rain layer obtained in the step S30 into the generating model of each iteration version obtained in the step S40, and carrying out quantitative and qualitative testing on the image to ensure that the generator G outputs a clean background image retaining the edge of the real defect and the texture detail so as to select a global optimal model.
Preferably, the step S10 is specifically implemented as follows:
firstly, capturing an original strip steel surface image moving at a high speed on a production line by using a high-speed camera to obtain a rainy layer-containing image and a clean background image. Then, aiming at different rain-like pseudo defects, a corresponding construction method is designed:
For dispersing water drops, in order to improve the generalization degree of the model, firstly, extracting real water drops from an original raindrop-like layer image, mixing the real water drops with artificially manufactured raindrops, and then pasting the mixture into a clean background image;
for a splash waterline, manufacturing rainwater-like lines with 4 inclinations and 6 aspect ratios by using Gaussian noise, and superposing the rainwater-like lines with a clean background image according to a certain proportion;
And for the fine white water drops, simulating the pixel level presentation of the fine white water drops generated during high-speed rolling of the steel belt in an original image, artificially simulating and manufacturing the fine white water drop pseudo defects with different sizes, and overlapping the fine white water drops with a clean background image.
Through the manufacturing process, a database containing 1450 pairs of 1000×1000 pixel hot rolled strip surface images is obtained, wherein one half of the database is an original clean image, and the other half of the database is a manually manufactured image with mixed rain-like pseudo defects. From which 1300 pairs of images are randomly drawn as the training set described in S30, leaving 150 pairs to be used as the test set described in S30.
Preferably, the specific substeps of step 20 are as follows:
Step S21: the training set described in step S30 is input in pairs into a generator, which selects a focus dual residual network model based on the encoder-decoder architecture. For the bottleneck layer, a periodic combined structure is proposed. It consists of one DuRB-P, one DuRB-DS with double SE modules and one DuRB-P connected end to end. The structure may be repeated three times in each iteration to progressively search for and recover rain-like artifacts, reducing the difference between a pair of images.
Step S22: in the training phase, a mask image is imported between the generator and the discriminator as a director to form a self-optimizing closed loop. The mask image generation method is a binary classification strategy based on a threshold value, and an equation can be expressed as follows:
Where Pixel rain-like layer is the Pixel value of the rain-like false defect image and Pixel clean is the Pixel value of the clean background image.
The director directs the generator to recover the local details of the mask by constructing a weighted sum of L1 and SSIM losses and considers global features to ensure that the generated image is free of distortion. The total loss value of L1 or SSIM is constructed as follows:
where loss_mask represents the Loss value of the rain-like layer, loss_Overall represents the Loss value of the entire image, and loss_Total represents the Total Loss of L1 or SSIM.
Step S23: a discriminator Res2Net with granular level multi-scale features is constructed, the input feature map is divided into four blocks after convolution, and the fidelity of the recovered features in the different blocks is discriminated. It can improve the multi-scale feature extraction capability without increasing the amount of computation, thereby guiding the generator in step S21 to restore the image texture more finely.
Preferably, the loss of the generator in step S40 is a weighted sum loss based on a fusion strategy. Structural Similarity (SSIM) can solve the distortion problem of images, and the SSIM loss function is expressed as follows:
Where o represents element multiplication and G represents a generation network. R and I respectively represent a clean background image and an image containing a rain-like layer, M represents a corresponding mask image, R-P clean represent R are pictures sampled from a clean background image sample, and I-P rain-like layer represent I are pictures sampled from an image sample containing a rain-like layer.
The L1 penalty function in the generator is as follows:
Finally, the loss function of the generator based on the fusion strategy is:
Where r 1 is set to 0.75, r 2 is set to 1.1, D is a discriminator, and V (G, D) is able to calculate JS divergence between the real image and the generated image.
The method for removing the rain-like layer on the surface of the hot rolled strip steel based on the attention dual residual error generation countermeasure network provided by the invention is compatible with channel attention and space attention, so that the position of the rain-like pseudo defect is accurately positioned and removed; meanwhile, the director introduced in the training stage skillfully digs the priori knowledge of the generalized rain-like layer to solve the problem of accidental misjudgment caused by the active searching strategy. Compared with a plurality of well-known rain removing algorithms, the method provided by the invention has the advantages that the image generated after the rain-like layer is removed is closest to the clean background image in the real industry, and the problem of serious false alarm of an AVI instrument caused by mixed rain-like pseudo defect clusters is effectively solved. Furthermore, the present invention contemplates a high resolution dataset for rain-like layer removal in automated steel surface inspection. This continuously disclosed dataset will motivate more rainy layer removal methods for industrial panel surface inspection in real world scenarios.
Drawings
FIG. 1 is a flow chart of a method for removing a rain-like layer on a surface image of a hot rolled strip steel based on an attention-dual residual error generation countermeasure network;
FIG. 2 is a schematic diagram of an overall model structure of a hot rolled strip steel surface image rain-like layer removing method based on an attention-dual residual error generation countermeasure network;
FIG. 3 is a schematic diagram of a model structure of a generator in a method for removing a rain-like layer on a surface image of a hot rolled strip steel based on an attention-dual residual generation countermeasure network;
FIG. 4 is a schematic diagram of a model structure of a discriminator in a method for removing a rain-like layer on a surface image of a hot rolled strip steel based on a focus dual residual error generation countermeasure network;
FIG. 5a is a drawing showing a real strip steel image containing a rain-like layer in an experimental example provided by the invention;
FIG. 5b is an image generated after the rain-like layer has been removed using ATTENTIVE GAN;
FIG. 5c is an image generated after removal of the rain-like layer using Pix2 Pix;
FIG. 5d is an image generated after the rain-like layer is removed using PReNet;
FIG. 5e is an image generated after the rain-like layer is removed using IADN;
FIG. 5f is an image generated after removal of a rain-like layer using DuRN-S-P;
FIG. 5g is an image generated after the rain-like layer has been removed using PReGAN;
FIG. 5h is an image generated after removing a rain-like layer by using the method for removing the rain-like layer of the surface image of the hot rolled strip steel based on the attention-dual residual error generation countermeasure network.
Detailed Description
The present invention will be described in detail below with reference to the accompanying drawings and examples. The following experimental examples and examples serve to further illustrate but not limit the invention.
Referring to fig. 1 to 4 in combination, the present invention provides a method for removing an image rain-like layer based on an attention dual residual generated countermeasure network, which is compatible with channel attention and space attention, and can accurately track and remove positions of pseudo defects of rain; meanwhile, the director introduced in the training stage skillfully digs the priori knowledge of the generalized rain-like layer to solve the problem of accidental misjudgment caused by the active searching strategy. The method provided by the invention can generate a clean background image closest to the real industry.
Specifically, the image rain-like layer removing method based on the attention-dual residual error generation countermeasure network provided by the invention comprises the following steps:
s10: aiming at three types of rain pseudo defects of scattered water drops, splashing water lines and tiny white water drops, a corresponding construction method is designed, and a rainy layer image database of the surface of the hot rolled strip steel is manufactured. The method is implemented by the following steps:
firstly, capturing an original strip steel surface image moving at a high speed on a production line by using a high-speed camera to obtain a rainy layer-containing image and a clean background image. Then, aiming at different rain-like pseudo defects, a corresponding construction method is designed:
For dispersing water drops, in order to improve the generalization degree of the model, firstly, extracting real water drops from an original raindrop-like layer image, mixing the real water drops with artificially manufactured raindrops, and then pasting the mixture into a clean background image;
for a splash waterline, manufacturing rainwater-like lines with 4 inclinations and 6 aspect ratios by using Gaussian noise, and superposing the rainwater-like lines with a clean background image according to a certain proportion;
And for the fine white water drops, simulating the pixel level presentation of the fine white water drops generated during high-speed rolling of the steel belt in an original image, artificially simulating and manufacturing the fine white water drop pseudo defects with different sizes, and overlapping the fine white water drops with a clean background image.
Through the manufacturing process, a database containing 1450 pairs of 1000×1000 pixel hot rolled strip surface images is obtained, wherein one half of the database is an original clean image, and the other half of the database is a manually manufactured image with mixed rain-like pseudo defects. From which 1300 pairs of images are randomly drawn as the training set described in S30, leaving 150 pairs to be used as the test set described in S30.
S20: the attention dual residual is built to generate an countermeasure network model, which consists of a generator G, a director M and a discriminator D. The specific substeps are as follows:
Step S21: the training set described in step S30 is input in pairs into a generator, which selects a focus dual residual network model based on the encoder-decoder architecture. The inputs include: the convolution layer, the batch standardized BN layer and the ReLU layer are sequentially connected, the structure is repeated three times, and one convolution layer is connected. For the bottleneck layer, a periodic combined structure is proposed. It consists of one DuRB-P, one DuRB-DS with double SE modules and one DuRB-P connected end to end. The structure may be repeated three times in each iteration to progressively search for and recover rain-like artifacts, reducing the difference between a pair of images. The receptive field of the convolution layer is continuously increased in 6 DuRB-P to fully utilize the context information of the image and increase the capability of multi-scale feature representation. The DuRB-DS configured with the dual SE modules has better global attention during the downsampling process, which helps infer the distribution of rain-like artifacts from real-world steel-band images. Referring to fig. 3, durb-P includes two convolution layers and two containers connected in sequence, the containers including one ConvLayers for placing the pair of operations. DuRB-DS comprises two convolutional layers and two containers connected in sequence, the containers comprising a ConvLayers convolutional layer module and a SE compression and excitation module (Squeeze and ExcitationModule).
Step S22: in consideration of the moire intensity information in the spatial dimension, a mask image is introduced between the generator and the discriminator as a director, thereby forming a self-optimizing closed loop in the training phase. The director utilizes potential prior knowledge of the generalized rain-like layer to solve the problems of edge blurring and detail loss caused by insufficient attention to the spatial features in the generator. The mask image generation method is a binary classification strategy based on a threshold value, and an equation can be expressed as follows:
Where Pixel rain-like layer is the Pixel value of the rain-like false defect image and Pixel clean is the Pixel value of the clean background image.
To ensure edge consistency of the recovery area, the director removes the rain-like layer by constructing a weighted sum supervision generator of L1 and SSIM losses. It can guide the generator to recover the local details of the mask and take global features into account to ensure that the generated image is not distorted. The total loss value of L1 or SSIM is constructed as follows:
where loss_mask represents the Loss value of the rain-like layer, loss_Overall represents the Loss value of the entire image, and loss_Total represents the Total Loss of L1 or SSIM.
Step S23: a discriminator Res2Net with granular level multi-scale features is constructed, the input feature map is divided into four blocks after convolution, and the fidelity of the recovered features in the different blocks is discriminated. It can improve the multi-scale feature extraction capability without increasing the amount of computation, thereby guiding the generator in step S21 to restore the image texture more finely. Referring to fig. 4, the discriminator includes 3 convolutional layers, 1 Res2Net layer, 1 fully-connected layer, and 1 Sigmoid layer, where Res2Net layer divides the feature map into four blocks x1, x2, x3, and x 4. Except for x1, all other blocks xi, i=2 to 4 need to go through the convolution layer, batch normalization layer and ReLU layer, and the result is defined as Ki (i.e., yi). The method comprises the specific steps that x2 is processed through a convolution layer, a batch standardization layer and a ReLU layer to obtain y2, a first result (K2) processed through the convolution layer, the batch standardization layer and the ReLU layer is processed through the convolution layer, the batch standardization layer and the ReLU layer together with x3 to obtain y3, a second result (K3) processed through the convolution layer, the batch standardization layer and the ReLU layer is processed through the convolution layer, the batch standardization layer and the ReLU layer together with x4 to obtain y4, and finally y1, y2, y3 and y4 are spliced and output.
S30: and (3) dividing the rainy layer image database manufactured in the step (S10) into a training set and a testing set.
S40: inputting the training set obtained in the step S30 into the attention dual residual error built in the step S20 in pairs to generate an countermeasure network model, introducing an attention scheme in the training process to mine priori knowledge of a generalized rain-like layer, repeatedly iterating and updating weights and losses of a generator and a discriminator, and obtaining a plurality of iteration version generation models.
Specifically, the loss of the generator in step S40 is a weighted sum loss based on the fusion policy. Structural Similarity (SSIM) can solve the distortion problem of images, and the SSIM loss function is expressed as follows:
Where o represents element multiplication and G represents a generation network. R and I respectively represent a clean background image and an image containing a rain-like layer, M represents a corresponding mask image, R-P clean represent R are pictures sampled from a clean background image sample, and I-P rain-like layer represent I are pictures sampled from an image sample containing a rain-like layer.
The L1 penalty function in the generator is as follows:
Finally, the loss function of the generator based on the fusion strategy is:
Where r 1 is set to 0.75, r 2 is set to 1.1, D is a discriminator, and V (G, D) is able to calculate JS divergence between the real image and the generated image.
S50: inputting the image of the testing set containing the rain layer obtained in the step S30 into the generating model of each iteration version obtained in the step S40, and carrying out quantitative and qualitative testing on the image to ensure that the generator G outputs a clean background image retaining the edge of the real defect and the texture detail so as to select a global optimal model.
As an experimental example of the invention, the self synthetic database is selected for training and testing, and meanwhile, several other well-known rainwater removal methods (ATTENTIVE GAN, pix2Pix, PReNet, IADN, duRN-S-P, PReGAN) are introduced as comparison groups, so that the superiority of the method is highlighted. The quantitative results are shown in FIGS. 5a to 5h.
The quantitative evaluation results of the different methods are summarized in table 1. These two indices illustrate that our proposed method can make the generated image more similar to a clean background image in the real industry. This is mainly because our method is more sensitive to the removal of rain-like layers, which uses channel attention and spatial attention to greatly influence the processing results without additional computational overhead.
Table 1 quantitative evaluation results of different methods.
The quantitative result of fig. 5 is shown in fig. 5a to 5h.
In FIG. 5, we compared the results of the present invention with those of ATTENTIVE GAN, pix2Pix, PReNet, IADN, durRN-S-P and PReGAN. FIG. 5a is an image of a real hot rolled strip containing a rain-like layer. Fig. 5b shows the image ATTENTIVE GAN generated with little or no large water droplets seen, but without overcoming the challenges presented by water lines and small white water droplets, and with global distortion. As can be seen from fig. 5c, pix2Pix cannot completely remove the rain-like layer although the generated image has no color difference. In addition, it removes some real defects, which will greatly affect the accuracy of subsequent defect detection. Fig. 5d is generated by PReNet with good global attention. However, in the case where the original image background is not bright, it cannot remove some minute water droplets. As shown in fig. 5e, IADN may remove most of the rain streaks while restoring the clearly visible image texture details, but still leave behind a large number of tiny rain-like artifacts. FIG. 5g, which is generated by DuRN-S-P and is shown in FIGS. 5f and PReGAN, also cannot remove fine white water droplets and water waves, so that the false detection rate of the AVI instrument is increased. As can be seen from fig. 5h, the method for removing the rain-like layer on the surface image of the hot rolled strip steel based on the attention dual residual error generation countermeasure network can remove extremely fine rain-like pseudo defects, and the generated image is more vivid.
The method for removing the rain-like layer on the surface of the hot rolled strip steel based on the attention dual residual error generation countermeasure network provided by the invention is compatible with channel attention and space attention, so that the position of the rain-like pseudo defect is accurately positioned and removed; meanwhile, the director introduced in the training stage skillfully digs the priori knowledge of the generalized rain-like layer to solve the problem of accidental misjudgment caused by the active searching strategy. Compared with a plurality of well-known rain removing algorithms, the method provided by the invention has the advantages that the image generated after the rain-like layer is removed is closest to the clean background image in the real industry, and the problem of serious false alarm of an AVI instrument caused by mixed rain-like pseudo defect clusters is effectively solved. Furthermore, the present invention contemplates a high resolution dataset for rain-like layer removal in automated steel surface inspection. This continuously disclosed dataset will motivate more rainy layer removal methods for industrial panel surface inspection in real world scenarios.
The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above examples, and all technical solutions belonging to the concept of the present invention belong to the protection scope of the present invention. It should be noted that numerous improvements and modifications can be made by those skilled in the art without departing from the principles of the present invention, and such improvements and modifications are also considered to be within the scope of the present invention.
Claims (3)
1. The method for removing the image rain-like layer based on the attention dual residual error to generate the countermeasure network comprises a rain-like pseudo defect construction method, a hot rolled strip steel surface rain-like layer image database and the attention dual residual error to generate the countermeasure network; the method for constructing the rain-like pseudo defect is used for restoring the shape and the distribution position of the mixed rain-like pseudo defect cluster in a real steel mill on a clean original strip steel image and manufacturing a hot-rolled strip steel surface rain-like layer image database; the hot rolled strip steel surface rain-like layer image database is used for training and testing analysis of the attention dual residual error generation countermeasure network model; the attention dual residual error generation countermeasure network model is used for removing rain-like pseudo defects on the strip steel image on the premise of retaining the real defect edges and texture details;
The method is implemented according to the following steps:
s10: aiming at three types of rain pseudo defects of scattered water drops, splashing water lines and tiny white water drops, a corresponding construction method is designed, and a rain-like layer image database on the surface of the hot-rolled strip steel is manufactured;
s20: building an attention dual residual error to generate an countermeasure network model, wherein the countermeasure network model consists of a generator G, a director M and a discriminator D;
The attention dual residual error generation countermeasure network model specifically comprises the following steps:
step S21: inputting the training set in step S30 into a generator, the generator selecting an attention-dual residual network model based on the encoder-decoder architecture; aiming at the bottleneck layer, a periodical combined structure is provided, which consists of a DuRB-P, a DuRB-DS with double SE modules and a DuRB-P end-to-end connection, wherein the structure is repeated three times in each iteration to gradually search and recover the rain-like pseudo defect, so that the difference between a pair of images is reduced;
step S22: in the training stage, a mask image is imported between a generator and a discriminator as a director to form a self-optimized closed loop, the mask image generation method is a binary classification strategy based on a threshold value, and an equation is expressed as follows:
wherein Pixelrain-LIKE LAYER are the pixel values of the rain-like false defect image and Pixelclean are the pixel values of the clean background image;
the director directs the generator to recover the local details of the mask by constructing a weighted sum of L1 and SSIM losses, and considers global features to ensure that the generated image is free of distortion; the total loss value of L1 or SSIM is constructed as follows:
where loss_mask represents the Loss value of the rain-like layer, loss_overall represents the Loss value of the entire image, and loss_total represents the Total Loss of L1 or SSIM;
Step S23: constructing a discriminator Res2Net with particle-level multi-scale features, dividing an input feature map into four blocks after convolution, distinguishing the fidelity degree of the restored features in different blocks, and improving the multi-scale feature extraction capability without increasing the calculation amount, so as to guide a generator in the step S21 to restore the image texture more finely;
S30: dividing the rainy layer-like image database manufactured in the step S10 into a training set and a testing set;
S40: inputting the training set obtained in the step S30 into the attention dual residual error built in the step S20 in pairs to generate an countermeasure network model, introducing an attention scheme in the training process to mine priori knowledge of a generalized rain-like layer, repeatedly iterating and updating weights and losses of a generator and a discriminator, and obtaining a plurality of iteration version generation models;
S50: inputting the rain-like layer-containing test set image obtained in the step S30 into the generation model of each iteration version obtained in the step S40, and carrying out quantitative and qualitative tests on the generation model to ensure that the generator G outputs a clean background image with the real defect edges and texture details reserved so as to select a global optimal model.
2. The method for removing the rain-like image based on the attention-dual residual generation countermeasure network according to claim 1, wherein the method for constructing the pseudo-rain defect and the database for the rain-like image on the surface of the hot-rolled strip steel in step S10 are specifically implemented according to the following steps:
Firstly, capturing an original strip steel surface image moving at a high speed on a production line by using a high-speed camera to obtain an image containing a rainy layer and a clean background image; then, aiming at different rain-like pseudo defects, a corresponding construction method is designed:
For dispersing water drops, in order to improve the generalization degree of the model, firstly, extracting real water drops from an original raindrop-like layer image, mixing the real water drops with artificially manufactured raindrops, and then pasting the mixture into a clean background image;
for a splash waterline, manufacturing rainwater-like lines with 4 inclinations and 6 aspect ratios by using Gaussian noise, and superposing the rainwater-like lines with a clean background image according to a certain proportion;
For the tiny white water drops, the tiny white water drops generated during high-speed rolling of the steel belt are simulated to be presented in the pixel level of the original image, and tiny white water drop pseudo defects with different sizes are manufactured through artificial simulation and are overlapped with a clean background image;
Through the manufacturing process, a database containing 1450 pairs of 1000×1000 pixel hot rolled strip steel surface images is obtained, wherein one half of the database is an original clean image, and the other half of the database is a manually manufactured image with mixed rain-like pseudo defects; from which 1300 pairs of images are randomly drawn as the training set described in S30, leaving 150 pairs to be used as the test set described in S30.
3. The method of removing rain layers from images based on attention-dual residuals to generate countermeasure networks according to claim 1, wherein the loss of the generator of step S40 is a weighted sum loss based on a fusion strategy; structural similarity SSIM solves the distortion problem of images, and the SSIM loss function is expressed as follows:
Wherein o represents element multiplication, G represents a generation network, R and I represent a clean background image and an image containing a rain-like layer, respectively, M represents a corresponding mask image, R-Pclean represent R are pictures sampled from a clean background image sample, and I-Prain-LIKE LAYER represent I are pictures sampled from an image sample containing a rain-like layer;
The L1 penalty function in the generator is as follows:
Finally, the loss function of the generator based on the fusion strategy is:
Where r1 is set to 0.75, r2 is set to 1.1, D is a discriminator, and V (G, D) is capable of calculating JS divergence between the true image and the generated image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210518394.4A CN114820379B (en) | 2022-05-12 | 2022-05-12 | Image rain-like layer removing method for generating countermeasure network based on attention dual residual error |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210518394.4A CN114820379B (en) | 2022-05-12 | 2022-05-12 | Image rain-like layer removing method for generating countermeasure network based on attention dual residual error |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114820379A CN114820379A (en) | 2022-07-29 |
| CN114820379B true CN114820379B (en) | 2024-04-26 |
Family
ID=82513426
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210518394.4A Active CN114820379B (en) | 2022-05-12 | 2022-05-12 | Image rain-like layer removing method for generating countermeasure network based on attention dual residual error |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114820379B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AU2020100196A4 (en) * | 2020-02-08 | 2020-03-19 | Juwei Guan | A method of removing rain from single image based on detail supplement |
| CN110992275A (en) * | 2019-11-18 | 2020-04-10 | 天津大学 | Refined single image rain removing method based on generation countermeasure network |
| CN111145112A (en) * | 2019-12-18 | 2020-05-12 | 华东师范大学 | Two-stage image rain removing method and system based on residual error countermeasure refinement network |
| CN112198170A (en) * | 2020-09-29 | 2021-01-08 | 合肥公共安全技术研究院 | Detection method for identifying water drops in three-dimensional detection of outer surface of seamless steel pipe |
| CN112258402A (en) * | 2020-09-30 | 2021-01-22 | 北京理工大学 | Dense residual generation countermeasure network capable of rapidly removing rain |
| CN113191969A (en) * | 2021-04-17 | 2021-07-30 | 南京航空航天大学 | Unsupervised image rain removing method based on attention confrontation generation network |
| KR102288645B1 (en) * | 2020-08-26 | 2021-08-10 | 한국해양과학기술원 | Machine learning method and system for restoring contaminated regions of image through unsupervised learning based on generative adversarial network |
| CN113450288A (en) * | 2021-08-04 | 2021-09-28 | 广东工业大学 | Single image rain removing method and system based on deep convolutional neural network and storage medium |
| CN113469913A (en) * | 2021-07-06 | 2021-10-01 | 中南大学 | Hot-rolled strip steel surface water drop removing method based on gradual cycle generation countermeasure network |
| CN114119382A (en) * | 2021-09-09 | 2022-03-01 | 浙江工业大学 | Image raindrop removing method based on attention generation countermeasure network |
| KR20220059881A (en) * | 2020-11-03 | 2022-05-10 | 고려대학교 산학협력단 | Progressive rain removal method and apparatus via a recurrent neural network |
-
2022
- 2022-05-12 CN CN202210518394.4A patent/CN114820379B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110992275A (en) * | 2019-11-18 | 2020-04-10 | 天津大学 | Refined single image rain removing method based on generation countermeasure network |
| CN111145112A (en) * | 2019-12-18 | 2020-05-12 | 华东师范大学 | Two-stage image rain removing method and system based on residual error countermeasure refinement network |
| AU2020100196A4 (en) * | 2020-02-08 | 2020-03-19 | Juwei Guan | A method of removing rain from single image based on detail supplement |
| KR102288645B1 (en) * | 2020-08-26 | 2021-08-10 | 한국해양과학기술원 | Machine learning method and system for restoring contaminated regions of image through unsupervised learning based on generative adversarial network |
| CN112198170A (en) * | 2020-09-29 | 2021-01-08 | 合肥公共安全技术研究院 | Detection method for identifying water drops in three-dimensional detection of outer surface of seamless steel pipe |
| CN112258402A (en) * | 2020-09-30 | 2021-01-22 | 北京理工大学 | Dense residual generation countermeasure network capable of rapidly removing rain |
| KR20220059881A (en) * | 2020-11-03 | 2022-05-10 | 고려대학교 산학협력단 | Progressive rain removal method and apparatus via a recurrent neural network |
| CN113191969A (en) * | 2021-04-17 | 2021-07-30 | 南京航空航天大学 | Unsupervised image rain removing method based on attention confrontation generation network |
| CN113469913A (en) * | 2021-07-06 | 2021-10-01 | 中南大学 | Hot-rolled strip steel surface water drop removing method based on gradual cycle generation countermeasure network |
| CN113450288A (en) * | 2021-08-04 | 2021-09-28 | 广东工业大学 | Single image rain removing method and system based on deep convolutional neural network and storage medium |
| CN114119382A (en) * | 2021-09-09 | 2022-03-01 | 浙江工业大学 | Image raindrop removing method based on attention generation countermeasure network |
Non-Patent Citations (3)
| Title |
|---|
| Progressive Image Deraining Networks: A Better and Simpler Baseline;Dongwei Ren等;《2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR)》;20200109;全文 * |
| 基于注意力生成对抗网络的图像超分辨率重建方法;丁明航;邓然然;邵恒;;计算机***应用;20200215(02);全文 * |
| 基于生成对抗网络去除单张图像中的雨滴;蒙佳浩;王东骥;帅天平;;软件;20200515(05);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114820379A (en) | 2022-07-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Tan et al. | Automatic detection of sewer defects based on improved you only look once algorithm | |
| CN110826588A (en) | Drainage pipeline defect detection method based on attention mechanism | |
| CN110020658B (en) | Salient object detection method based on multitask deep learning | |
| CN114742799B (en) | Industrial scene unknown type defect segmentation method based on self-supervision heterogeneous network | |
| CN112633149A (en) | Domain-adaptive foggy-day image target detection method and device | |
| CN115131283A (en) | Defect detection and model training method, device, equipment and medium for target object | |
| CN111209858A (en) | Real-time license plate detection method based on deep convolutional neural network | |
| Lai et al. | Industrial anomaly detection and one-class classification using generative adversarial networks | |
| Li et al. | A review of deep learning methods for pixel-level crack detection | |
| Guo et al. | Joint raindrop and haze removal from a single image | |
| Guo et al. | A novel transformer-based network with attention mechanism for automatic pavement crack detection | |
| Qian et al. | FAOD‐Net: a fast AOD‐Net for dehazing single image | |
| CN116030361A (en) | CIM-T architecture-based high-resolution image change detection method | |
| CN113421223B (en) | Industrial product surface defect detection method based on deep learning and Gaussian mixture | |
| Xu et al. | Multiple guidance network for industrial product surface inspection with one labeled target sample | |
| CN117372876A (en) | Road damage evaluation method and system for multitasking remote sensing image | |
| CN113516652A (en) | Battery surface defect and adhesive detection method, device, medium and electronic equipment | |
| CN117557784A (en) | Target detection method, target detection device, electronic equipment and storage medium | |
| CN114820379B (en) | Image rain-like layer removing method for generating countermeasure network based on attention dual residual error | |
| CN111612803B (en) | Vehicle image semantic segmentation method based on image definition | |
| CN116630989A (en) | Visual fault detection method and system for intelligent ammeter, electronic equipment and storage medium | |
| Mi et al. | Dense residual generative adversarial network for rapid rain removal | |
| CN105354833A (en) | Shadow detection method and apparatus | |
| CN113192018B (en) | Water-cooled wall surface defect video identification method based on fast segmentation convolutional neural network | |
| CN112001396B (en) | Bearing surface deformation and character mixed defect image detection method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |