CN111553856B - Image defogging method based on depth estimation assistance - Google Patents
Image defogging method based on depth estimation assistance Download PDFInfo
- Publication number
- CN111553856B CN111553856B CN202010331563.4A CN202010331563A CN111553856B CN 111553856 B CN111553856 B CN 111553856B CN 202010331563 A CN202010331563 A CN 202010331563A CN 111553856 B CN111553856 B CN 111553856B
- Authority
- CN
- China
- Prior art keywords
- layer
- image
- convolution unit
- convolution
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000012549 training Methods 0.000 claims abstract description 24
- 238000004422 calculation algorithm Methods 0.000 claims description 40
- 238000011176 pooling Methods 0.000 claims description 32
- 230000004913 activation Effects 0.000 claims description 10
- 239000011159 matrix material Substances 0.000 claims description 8
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 6
- 238000002834 transmittance Methods 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 4
- 238000012805 post-processing Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 239000003595 mist Substances 0.000 claims 1
- 238000006467 substitution reaction Methods 0.000 claims 1
- 238000012545 processing Methods 0.000 abstract description 5
- 230000000694 effects Effects 0.000 description 12
- 238000004088 simulation Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 102100040954 Ephrin-A1 Human genes 0.000 description 7
- 101000965523 Homo sapiens Ephrin-A1 Proteins 0.000 description 7
- 238000013527 convolutional neural network Methods 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 5
- 230000004927 fusion Effects 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000010521 absorption reaction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000005352 clarification Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an image defogging method based on depth estimation assistance, which mainly solves the problems of poor fog distribution estimation and loss of recovered image texture details in the prior art. The scheme is as follows: respectively constructing a depth estimation network and a defogging network under a Pythrch frame; obtaining a group of fog-free image sets J, and performing depth estimation and artificial fog adding on the J to obtain a depth image set D and a fog image set I; respectively training a depth estimation network and a defogging network by using the depth image set and the foggy image set to obtain the trained depth estimation network and defogging network; image I to be defogged c Inputting the depth value to a trained depth estimation network, and outputting an estimated depth value D c (ii) a Image I to be defogged c And depth value D c Inputting the image into a trained defogging network and outputting a clear image. The method can well restore the details and the tone of the image, has the peak signal-to-noise ratio and the structural similarity higher than or close to those of the prior art, and can be used for the clear processing of the foggy image.
Description
Technical Field
The invention belongs to the technical field of digital image processing, and particularly relates to an image defogging method which can be used for the clarification of a single foggy image shot by an imaging system in a foggy environment.
Background
The scattering and absorption of media are common reasons causing image degradation, under the influence of haze, image acquisition equipment is affected by the scattering and absorption of a large number of suspended particles in the air to light, and a shot image is low in visibility and contrast and cannot be distinguished by partial objects. Thereby severely affecting the effectiveness of various types of image-based information processing systems. Therefore, the research on the image defogging under the haze condition has important significance and application prospect.
Early defogging algorithms were mostly based on prior knowledge and atmospheric scattering models, and the key issue was how to estimate atmospheric light and transmittance. The method takes the forming principle of the fog image as a basis, extracts the characteristics of the fog image through various prior assumptions, estimates the parameters in the atmospheric scattering model by using the characteristics, and substitutes the parameters into a model formula to realize image defogging. Known Dark channel prior defogging methods (DCP) as proposed by He et al, see He K, SUN J, TANG x.single image size removal using Dark channel prior [ J ]// IEEE transaction pattern analysis and machine interaction, 2011,33 (12): 2341-2353.
With the rise of deep learning technology, the defogging method based on deep learning gradually becomes a research hotspot in the field of image defogging at present. The convolutional neural network is applied to the field of Image defogging, namely a DehazeNet network, which is proposed by Cai et al, and is shown in CAI B, XU X, JIAK, et al. Then Ren et al proposed a multi-scale estimation algorithm based on convolutional neural networks to estimate the transmittance in the atmospheric scattering model — MSCNN network, see Ren W, LIU S, ZHANG H, et al.
In order to solve the color difference caused by inaccurate atmospheric light estimation and the attenuation phenomenon of light in the transmission process, most of the defogging methods which appear recently uniformly adopt a convolution neural network to directly estimate the defogged image, and realize the end-to-end processing from the defogging image to the defogging image. A typical method comprises:
1) Ren et al proposed a door Fusion defogging Network in 2018, see Wenqi Ren, lin Ma, jiawei Zhang, jin shan Pan, et al, door Fusion Network for Single Image Dehazing [ C ]// Proceedings of the IEEE Conference on Computer Vision and Pattern recognition.2018, 3253-3261. It first performs preprocessing such as white balance, contrast enhancement and gamma enhancement on a foggy Image, and then performs weighted Fusion with three feature maps output by the Network respectively to finally obtain a defogged Image. (ii) a
2) Liu et al propose a common Model-Agnostic Convolutional Neural Network for defogging images, see [18] Z.Liu, B.Xiao, M.Alrabeiah, K.Wang and J.Chen, single Image Dehazing with a Generic Model-adaptive computational consistent Network [ J ]// Signal Processing Letters,2019,26 (6): 833-837. This method designs an end-to-end defogging Network on the basis of a coding-decoding frame without the aid of an atmospheric imaging Model, extracts the generalized features of a foggy day Image by a residual module, which has a good recovery effect on the foggy Image, but because its Network employs an deconvolution layer, the recovered Image usually has grid-like subjective stripes, affecting the visual perception.
In summary, the existing end-to-end defogging neural network has the following disadvantages: firstly, the door fusion network is not designed with an estimation structure aiming at fog distribution, so that the phenomenon of incomplete defogging occurs; secondly, the unknown convolution neural network model is simple in structure, adopts deconvolution layers, is restrained in nonlinear fitting capacity and is prone to chessboard effect.
Disclosure of Invention
The invention aims to provide an image defogging method based on depth estimation assistance aiming at the defects of the prior art, so as to completely utilize a convolutional neural network to perform defogging, reduce the loss of recovered image texture details and improve the defogging effect.
In order to achieve the above object, the depth-aided estimation image defogging method of the invention comprises the following steps:
1) Respectively constructing a depth estimation network and a defogging network under a Pythrch frame:
the depth estimation network comprises eight convolution units, six residual error module layers, six upsampling layers and five pooling layers; the defogging network comprises five convolution units, six recalibration module layers, an upper sampling layer and five pooling layers;
each convolution unit comprises a convolution layer and a normalization layer, wherein the convolution layer comprises a convolution operation layer and a LeakyReLU activation function layer;
2) Obtaining a set of clear fog-free image sets J t And depth image set D t From a depth image set D t Calculating to obtain a transmissivity image set T, and carrying out fog-free image J t Carrying out artificial fogging according to the transmissivity image set T and the artificially set atmospheric light value A to obtain a foggy image set I t Set of fog images I t And depth image set D t As a training image set of a depth estimation network, a fog-free image set J is adopted t And fog image set I t As a training image set of the defogging network;
3) Training the depth estimation network:
3a) Will train the image set I t Each image in the depth estimation network is sequentially input into the depth estimation network, and the estimated corresponding depth image is output
3b) The depth image D corresponding to the depth image set t Depth image corresponding to estimated
Substituting the weighted total loss formula into the weighted total loss formula, and calculating a weighted total loss value corresponding to each image in the training set;
3c) Updating a direction formula by using network parameters of an Adam algorithm, taking the minimum weighted total loss value as a target, and updating parameters in the depth estimation network to obtain a trained depth estimation network;
4) Training the defogging network:
4a) Will train the image set I t Each image in the depth estimation system is sequentially input into a trained depth estimation network, and an estimated corresponding depth image D is output * ；
4b) Will train the image set I t Each image in (a) and its corresponding depth image D * Inputting the image into defogging network, and outputting corresponding estimated defogged image J * ；
4c) Concentrating the corresponding fog-free image J t Defogged image J corresponding to the estimated * Substituting the weighted total loss formula into the weighted total loss formula, and calculating a weighted total loss value corresponding to each image in the training set;
4e) Updating a direction formula by using network parameters of an Adam algorithm, taking the minimum weighted total loss value as a target, and updating parameters in a depth estimation network to obtain a trained defogging network;
5) A foggy image I needing the defogging treatment is put into c Inputting the depth image into a trained depth estimation network and outputting a depth image D c ；
6) Depth image D to be output c And a foggy image I c Inputting the images into a trained defogging network, and outputting a defogged image J c 。
Compared with the prior art, the invention has the following beneficial effects:
firstly, the depth of the image is estimated in advance through a depth estimation network, so that the problem of incomplete defogging in the edge region of the scene in the prior art is avoided;
second, the invention provides better preservation of detail information of restored images due to updating of network parameters based on weighted total loss for texture details.
Simulation results show that the method has an obvious defogging effect compared with the two methods of the existing door fusion network and the model-agnostic convolution neural network on the premise of keeping the contrast and the color saturation of the recovered image, can better recover the background information of the image and improve the visual effect; and the peak signal-to-noise ratio PNSR is superior to that of the prior art, and the structural similarity SSIM is close to that of the prior art.
Drawings
FIG. 1 is a general flow chart of an implementation of the present invention;
FIG. 2 is a diagram of a depth estimation network architecture constructed in accordance with the present invention;
FIG. 3 is a diagram of a defogging network constructed in the present invention;
FIG. 4 is a graph comparing the defogging effects of a synthesized fog image by using the present invention and a prior deep learning defogging algorithm;
fig. 5 is a comparison graph of defogging effects of the real fog images by using the defogging algorithm based on the deep learning of the invention and the defogging algorithm based on the deep learning.
Detailed Description
The following further describes embodiments of the present invention with reference to the accompanying drawings:
referring to fig. 1, the specific implementation steps of the present invention are as follows:
step 1: and constructing a depth estimation network architecture under a Pythrch framework.
As shown in fig. 2, the depth estimation network constructed under the Pytorch framework of the present invention includes a pre-processing portion, a post-processing portion, and an output portion, and sets eight convolution units, six residual module layers, six upsampling layers, and five pooling layers, where each convolution unit includes a convolution layer and a normalization layer, and the convolution layer includes a convolution operation and a leakage relu activation function layer.
The pretreatment part sequentially comprises the following structures: the first convolution unit → the second convolution unit → the third convolution unit → the first residual module layer → the first pooling layer → the second residual module layer → the second pooling layer → the third residual module layer → the third pooling layer → the fourth residual module layer → the first upsampling layer → the fifth residual module layer → the second upsampling layer → the sixth residual module layer → the third upsampling layer; the first residual error module layer is simultaneously connected with the sixth residual error module layer; the second residual error module layer is simultaneously connected with the fifth residual error module layer; the third residual module layer is simultaneously connected with the fourth residual module layer;
the post-processing part comprises three branch structures: the first branch is in turn: the fourth pooling layer → the fourth convolution unit → the fourth upsampling layer; the second branch is sequentially as follows: fifth pooling layer → fifth convolution unit → fifth upsampling layer; the third branch is sequentially as follows: sixth pooling layer → sixth convolution unit → sixth upsampling layer; the fourth pooling layer, the fifth pooling layer and the sixth pooling layer are all connected with the third upper sampling layer;
the structure of the output part is as follows in sequence: seventh convolution unit → eighth convolution unit; the seventh convolution unit is simultaneously connected with the fourth upsampling layer, the fifth upsampling layer and the sixth upsampling layer.
The parameters of each layer are set as follows:
the convolution kernels of the first convolution unit and the second convolution unit are 7 × 7 and 5 × 5 respectively;
the sizes of convolution kernels of the third convolution unit, the fourth convolution unit, the fifth convolution unit, the sixth convolution unit, the seventh convolution unit and the eighth convolution unit are all 3 x 3; the convolution step length of all convolution units is 1;
the scaling factors of the first, second and third pooling layers are all 2;
the scaling factor of the fourth pooling layer is 4, the scaling factor of the fifth pooling layer is 8, and the scaling factor of the sixth pooling layer is 16;
the scaling factors of the first upsampling layer, the second upsampling layer and the third upsampling layer are all 2;
the fourth upsampling layer has a scaling factor of 4, the fifth upsampling layer has a scaling factor of 8, and the sixth upsampling layer has a scaling factor of 16.
Step 2: and constructing a defogging network architecture under a Pythrch framework.
As shown in fig. 3, the defogging network constructed in this example includes an encoding portion and a decoding portion, and has five convolution units, six recalibration module layers, one upsampling layer, and five pooling layers, each convolution unit including a convolution layer and a normalization layer, the convolution layer including a convolution operation and a leakage relu activation function layer. Each recalibration module layer comprises two parts: the first part is as follows: the first convolution unit → the second convolution unit → the first full connection layer → LeakyReLU activation function layer → the second full connection layer → Sigmod activation function layer; the second part is as follows in sequence: III convolution unit → I upsampling layer → IV convolution unit. Connecting the Sigmod activation function layer with a third convolution unit, and connecting a second convolution unit with a third convolution unit, wherein the parameters of the recalibration module layer are set as follows: the convolution kernels of the first convolution unit, the second convolution unit, the third convolution unit and the fourth convolution unit are all 3 x 3; the convolution step length of the first convolution unit is 2; the convolution step lengths of the second convolution unit, the third convolution unit and the fourth convolution unit are all 1; the scaling factor of the I upsampling layer is 2; the number of input feature maps and the number of output feature maps of the first fully-connected layer are respectively 64 and 32; the number of input profiles and the number of output profiles for the IIth fully-connected layer are 32 and 64, respectively.
The structure of the coding part is as follows in sequence: the 1 st convolution unit → the 2 nd convolution unit → the 3 rd convolution unit → the 4 th convolution unit → the 1 st retargeted module layer → the 2 nd retargeted module layer → the 3 rd retargeted module layer;
the structure of the decoding part is as follows in sequence: the 4 th recalibration module layer → the 5 th recalibration module layer → the 6 th recalibration module layer → the 1 st upper sampling layer → the 5 th convolution unit;
the 4 th convolution unit is simultaneously connected with the 2 nd re-calibration module layer, the 3 rd re-calibration module layer, the 4 th re-calibration module layer, the 5 th re-calibration module layer and the 6 th re-calibration module layer respectively.
The parameters of each layer are set as follows:
the convolution kernel size of the 1 st convolution unit is 7 × 7, and the convolution kernel size of the 2 nd convolution unit is 5 × 5;
the convolution kernels of the 3 rd convolution unit, the 4 th convolution unit and the 5 th convolution unit are all 3 x 3 in size;
the convolution step lengths of the 1 st convolution unit, the 2 nd convolution unit and the 3 rd convolution unit and the 5 th convolution unit are all 1;
the convolution step of the 4 th convolution unit is 2;
the scaling factor for the 1 st upsampling layer is 2.
And step 3: and (5) making a training image set.
3.1 ) download 1000 fog-free image sets J of different scenes from the network t Using a bilinear interpolation algorithm to interpolate J t Is uniformly scaled to 256 × 256;
3.2 For fog-free image set J t Respectively estimating depth information D corresponding to each image by using a depth-of-field estimation CNN model t ；
3.3 Using random function to randomly generate a transmittance parameter beta between 1.0 and 2.5, and calculating the transmittance of each graph
3.4 Using random function to randomly generate atmospheric light value A between 0.8-1.0, and calculating to obtain 20000 fog images I t ＝J t T + A (1-T) as a set of foggy images;
3.5 Will have a fog image set I t And depth information D t A training image set as a depth estimation network;
3.6 Set J of fog-free images t And fog image set I t As a training image set for the defogging network.
And 4, step 4: and training the deep estimation network.
4.1 A set of training images I) t Each image in the depth estimation network is input into the depth estimation network in sequence, and the estimated corresponding depth image is output
4.2 From corresponding depth images D in the depth image set t And estimated corresponding depth image
And calculating the weighted total loss value of the deep training set by the following formula:
wherein | | 2 To perform a two-norm operation on the matrix, | - | is an absolute value operation on the matrix, m is the number of pixels of the input image, and Sobel (·) is a convolution operation using a four-direction Sobel operator.
The calculation of the weighted total loss value for the deep training set is not limited to the above formula.
4.3 Using a network parameter updating direction formula of an Adam algorithm, updating parameters of the depth estimation network by taking the minimum weighted total loss value as a target, and repeating the steps 4.1), 4.2) and 4.3) 10000000 times to obtain the trained depth estimation network.
And 5: and training the defogging network.
5.1 Estimate depth image):
will train the image set I t Each image in the depth estimation network is input into the trained depth estimation network in sequence, and the estimated corresponding depth image is output
5.2 Estimate a defogged image:
will train the image set I t Each image in (1) and its corresponding depth image
Inputting the image into a defogging network, and outputting the corresponding estimated defogged image->
5.3 Based on the fog-free image set corresponding fog-free image J t And the estimated corresponding defogged image
And calculating the weighted total loss value of the defogging training set by the following formula:
wherein | - | 2 To perform a two-norm operation on the matrix, | - | is an absolute value operation on the matrix, m is the number of pixels of the input image, and Sobel (·) is a convolution operation using a four-direction Sobel operator.
The calculation of the weighted total loss value for the defogging training set is not limited to the above formula.
5.4 The direction formula is updated by using the network parameters of the Adam algorithm, the parameters of the defogging network are updated by taking the minimum weighted total loss value as a target, and the steps 5.1), 5.2), 5.3) and 5.4) are repeated for 8000000 times, so that the trained defogging network is obtained.
Step 6: and carrying out image defogging.
6.1 A foggy image I that will require defogging processing c Inputting a trained depth estimation network, and outputting a depth image D by the network c ；
6.2 Will have a fog image I c And depth image D c Inputting the trained defogging network together, and outputting a clear image J through the network c 。
The effects of the present invention are further illustrated by the following simulations:
simulation conditions
1. Testing pictures: four real fog maps and Reside OTS and Make3D data sets downloaded from the network;
2. the simulation method comprises the following steps: the existing DCP algorithm, the DehazeNet algorithm, the GMAN algorithm, the DuRB-US algorithm and the invention are used for five methods;
secondly, simulation test content:
simulation test 1: the four resultant fog maps were defogged using the five methods described above, the results are shown in fig. 4, where:
figure 4a is a composite four hazy image,
figure 4b is the result of the defogging process using the DCP algorithm on the foggy image of figure 4a,
figure 4c is the result of the defogging process using the DehazeNet algorithm on the foggy image of figure 4a,
figure 4d is the result of defogging the foggy image of figure 4a using the GMAN algorithm,
figure 4e is the result of the defogging process using the DuRB-US algorithm on the foggy image of figure 4a,
figure 4f is the result of the defogging process applied to the foggy image of figure 4a using the method of the present invention,
FIG. 4g is four fog-free images;
as can be seen from fig. 4, the images restored by using the existing DCP algorithm and the DuRB-US algorithm have a certain color cast, and the images restored by using the existing DehazeNet algorithm and the GMAN algorithm still have residual fog. The image effect recovered by the method is better than other four defogging algorithms and is closer to 4g of a fog-free image.
Simulation test 2: the five methods are used for carrying out defogging treatment on the four real fog images respectively, and the effect is shown in fig. 5, wherein:
figure 5a shows four true fogging images,
figure 5b is the result of the defogging process using the DCP algorithm on the foggy image of figure 5a,
figure 5c is the result of the defogging process using the DehazeNet algorithm on the foggy image of figure 5a,
figure 5d is the result of defogging the fogged image of figure 5a using the GMAN algorithm,
figure 5e is the result of the defogging process using the DuRB-US algorithm on the foggy image of figure 5a,
FIG. 5f is the result of defogging the image of FIG. 5a using the method of the present invention;
as can be seen from fig. 5, the images restored by using the existing DCP algorithm and the DuRB-US algorithm have severe color cast, and the images restored by using the existing DehazeNet algorithm and the GMAN algorithm still have obvious haze in the edge region of the object. The image effect recovered by the method of the invention is better than other four defogging algorithms.
Simulation test 3: defogging treatment is carried out on the Reside OTS and Make3D data sets by using the five methods, and the structural similarity SSIM index and the peak signal-to-noise ratio PNSR index are compared, as shown in Table 1:
TABLE 1
From table 1, the PSNR and SSIM values of the method of the present invention are higher than or equal to those of the three algorithms DCP, dehazeNet, and DuRB-US, while the SSIM index is slightly lower than that of the GMAN algorithm, the PSNR index is greatly higher than that of the GMAN algorithm, and the overall effect is better than that of the other four algorithms.
By combining the comparison of the simulation results of the five algorithms, the defogging effect of the method on various foggy images is superior to that of other four algorithms.
Claims (10)
1. An image defogging method based on depth estimation assistance is characterized by comprising the following steps:
1) Respectively constructing a depth estimation network and a defogging network under a Pythrch framework:
the depth estimation network comprises eight convolution units, six residual error module layers, six upsampling layers and five pooling layers; the defogging network comprises five convolution units, six recalibration module layers, an upper sampling layer and five pooling layers;
each convolution unit comprises a convolution layer and a normalization layer, wherein the convolution layer comprises a convolution operation layer and a LeakyReLU activation function layer;
2) Obtaining a set of clear fog-free image sets J t Respectively estimating depth-of-field information D corresponding to each image by using a depth-of-field estimation CNN model t According to depthImage set D t Calculating to obtain a transmissivity image set T, and carrying out fog-free image J t Carrying out artificial fogging according to the transmissivity image set T and the artificially set atmospheric light value A to obtain a foggy image set I t Set of fog images I t And depth image set D t As a training image set of a depth estimation network, a fog-free image set J is adopted t And fog image set I t As a training image set of the defogging network;
3) Training the depth estimation network:
3a) Will train the image set I t Each image in the image data is sequentially input into a depth estimation network, and the estimated corresponding depth is output
Image of a person
3b) The depth image D corresponding to the depth image set t Depth image corresponding to the estimated
Substituting the weighted total loss formula designed for the texture details, and calculating the weighted total loss value corresponding to each image in the training set;
3c) Updating a direction formula by using network parameters of an Adam algorithm, taking the minimum weighted total loss value as a target, and updating parameters in the depth estimation network to obtain a trained depth estimation network;
4) Training the defogging network:
4a) Will train the image set I t Each image in the depth estimation system is sequentially input into a trained depth estimation network, and an estimated corresponding depth image D is output * ；
4b) Will train the image set I t Each image in (a) and its corresponding depth image D * Inputting into a defogging network, and outputting the corresponding estimated defogged image J * ；
4c) Concentrating the corresponding fog-free image J t Defogged image J corresponding to the estimated * Substitution is fine for textureCalculating a weighted total loss value corresponding to each image in a training set according to a designed weighted total loss formula;
4e) Updating a direction formula by using network parameters of an Adam algorithm, taking the minimum weighted total loss value as a target, and updating parameters in a depth estimation network to obtain a trained defogging network;
5) A foggy image I needing the defogging treatment is put into c Inputting the depth image into a trained depth estimation network and outputting a depth image D c ；
6) Depth image D to be output c And a foggy image I c Inputting the defogging network into the training network and outputting a defogged image J c 。
2. The method of claim 1, wherein: 1) The depth estimation network constructed in (1) comprises a pre-processing part, a post-processing part and an output part;
the pretreatment part comprises the following steps: the first convolution unit → the second convolution unit → the third convolution unit → the first residual module layer → the first pooling layer → the second residual module layer → the second pooling layer → the third residual module layer → the third pooling layer → the fourth residual module layer → the first upsampling layer → the fifth residual module layer → the second upsampling layer → the sixth residual module layer → the third upsampling layer; the first residual error module layer is simultaneously connected with the sixth residual error module layer; the second residual module layer is simultaneously connected with the fifth residual module layer, and the third residual module layer is simultaneously connected with the fourth residual module layer;
the post-processing section has three branches:
the first branch is in turn: a fourth pooling layer → a fourth convolution unit → a fourth upsampling layer;
the second branch is sequentially as follows: fifth pooling layer → fifth convolution unit → fifth upsampling layer;
the third branch is sequentially as follows: sixth pooling layer → sixth convolution unit → sixth upsampling layer; the fourth pooling layer, the fifth pooling layer and the sixth pooling layer are all connected with the third upper sampling layer;
the output part is sequentially as follows: seventh convolution unit → eighth convolution unit; the seventh convolution unit is simultaneously connected with the fourth upsampling layer, the fifth upsampling layer and the sixth upsampling layer.
3. The method of claim 2, wherein: the parameters of each layer of the depth estimation network are set as follows:
the convolution kernels of the first convolution unit and the second convolution unit are 7 × 7 and 5 × 5 respectively;
the sizes of convolution kernels of the third convolution unit, the fourth convolution unit, the fifth convolution unit, the sixth convolution unit, the seventh convolution unit and the eighth convolution unit are all 3 x 3; the convolution step length of all convolution units is 1;
the scaling factors of the first, second and third pooling layers are all 2;
the scaling factor of the fourth pooling layer is 4, the scaling factor of the fifth pooling layer is 8, and the scaling factor of the sixth pooling layer is 16;
the scaling factors of the first upsampling layer, the second upsampling layer and the third upsampling layer are all 2;
the scaling factor for the fourth upsampling layer is 4, the scaling factor for the fifth upsampling layer is 8, and the scaling factor for the sixth upsampling layer is 16.
4. The method of claim 1, wherein: 1) The defogging network constructed in (1), which includes an encoding portion and a decoding portion;
the coding part is sequentially as follows: the 1 st convolution unit → the 2 nd convolution unit → the 3 rd convolution unit → the 4 th convolution unit → the 1 st retargeted module layer → the 2 nd retargeted module layer → the 3 rd retargeted module layer;
the decoding part is sequentially as follows: the 4 th recalibration module layer → the 5 th recalibration module layer → the 6 th recalibration module layer → the 1 st upper sampling layer → the 5 th convolution unit;
and simultaneously and respectively connecting the 4 th convolution unit with the 2 nd re-calibration module layer, the 3 rd re-calibration module layer, the 4 th re-calibration module layer, the 5 th re-calibration module layer and the 6 th re-calibration module layer to obtain the constructed defogging network.
5. The method of claim 4, wherein: the parameters of each layer of the defogging network are set as follows:
the convolution kernel size of the 1 st convolution unit is 7 × 7, and the convolution kernel size of the 2 nd convolution unit is 5 × 5;
the convolution kernels of the 3 rd convolution unit, the 4 th convolution unit and the 5 th convolution unit are all 3 x 3 in size;
the convolution step lengths of the 1 st convolution unit, the 2 nd convolution unit and the 3 rd convolution unit and the 5 th convolution unit are all 1;
the convolution step of the 4 th convolution unit is 2;
the scaling factor for the 1 st upsampling layer is 2.
6. The method of claim 1, wherein: 1) Each recalibration module layer in (2) has a structure of two parts:
the first part is as follows: the first convolution unit → the second convolution unit → the first fully connected layer → the LeakyReLU activation function layer → the second fully connected layer → the Sigmod activation function layer;
the second part is as follows in sequence: the III convolution unit → the I upsampling layer → the IV convolution unit; and connecting the Sigmod activation function layer with the III convolution unit, and connecting the II convolution unit with the III convolution unit to obtain the recalibration module layer.
7. The method of claim 6, wherein: the parameters of the recalibration module layer are set as follows:
the convolution kernels of the first convolution unit, the second convolution unit, the third convolution unit and the fourth convolution unit are all 3 x 3;
the convolution step length of the first convolution unit is 2;
the convolution step lengths of the second convolution unit, the third convolution unit and the fourth convolution unit are all 1;
the scaling factor of the I upsampling layer is 2;
the number of input feature maps and the number of output feature maps of the first fully-connected layer are respectively 64 and 32;
the number of input profiles and the number of output profiles for the IIth fully-connected layer are 32 and 64, respectively.
8. The method of claim 1, wherein: 2) According to depth image set D t The transmittance image set T is obtained through calculation, and the transmittance image set T is calculated through the following formula:
wherein e is the base of the logarithm of the natural number, β is the atmospheric scattering coefficient, where it is assumed that the mist is uniformly distributed in the image, β is randomly generated between 1.0 and 2.5 using random function, D t For corresponding depth image sets D t 。
9. The method of claim 1, wherein the weighted total loss formula in 3 b) is expressed as follows:
wherein | | 2 For a two-norm operation on a matrix, | - | is an absolute value operation on the matrix, m is the number of pixels of the input image, sobel (. -) is a convolution operation using a four-way Sobel operator, D t For a corresponding depth image in the depth image set,
is the estimated corresponding depth image.
10. The method of claim 1, wherein the weighted total loss formula in 4 c) is expressed as follows:
wherein | - | 2 For a two-norm operation on a matrix, | - | is an absolute value operation on the matrix, m is the number of pixels of the input image, sobel (. -) is a convolution operation using a four-way Sobel operator, J t Is a corresponding fog-free image in the fog-free image set,
is the estimated corresponding defogged image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010331563.4A CN111553856B (en) | 2020-04-24 | 2020-04-24 | Image defogging method based on depth estimation assistance |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010331563.4A CN111553856B (en) | 2020-04-24 | 2020-04-24 | Image defogging method based on depth estimation assistance |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111553856A CN111553856A (en) | 2020-08-18 |
| CN111553856B true CN111553856B (en) | 2023-03-24 |
Family
ID=72003942
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010331563.4A Active CN111553856B (en) | 2020-04-24 | 2020-04-24 | Image defogging method based on depth estimation assistance |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111553856B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112365476B (en) * | 2020-11-13 | 2023-12-08 | 南京信息工程大学 | Fog day visibility detection method based on double-channel depth network |
| CN112734679A (en) * | 2021-01-26 | 2021-04-30 | 西安理工大学 | Fusion defogging method for medical operation video images |
| CN115883851A (en) * | 2021-09-28 | 2023-03-31 | 腾讯科技（深圳）有限公司 | Filtering, encoding and decoding methods and devices, computer readable medium and electronic equipment |
| CN114255178A (en) * | 2021-12-03 | 2022-03-29 | 天津大学 | Video defogging method based on depth information guidance |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108805839A (en) * | 2018-06-08 | 2018-11-13 | 西安电子科技大学 | Combined estimator image defogging method based on convolutional neural networks |
| CN109146810A (en) * | 2018-08-08 | 2019-01-04 | 国网浙江省电力有限公司信息通信分公司 | A kind of image defogging method based on end-to-end deep learning |
| CN109272455A (en) * | 2018-05-17 | 2019-01-25 | 西安电子科技大学 | Based on the Weakly supervised image defogging method for generating confrontation network |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8350933B2 (en) * | 2009-04-08 | 2013-01-08 | Yissum Research Development Company Of The Hebrew University Of Jerusalem, Ltd. | Method, apparatus and computer program product for single image de-hazing |
| CN106157307B (en) * | 2016-06-27 | 2018-09-11 | 浙江工商大学 | A kind of monocular image depth estimation method based on multiple dimensioned CNN and continuous CRF |
-
2020
- 2020-04-24 CN CN202010331563.4A patent/CN111553856B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109272455A (en) * | 2018-05-17 | 2019-01-25 | 西安电子科技大学 | Based on the Weakly supervised image defogging method for generating confrontation network |
| CN108805839A (en) * | 2018-06-08 | 2018-11-13 | 西安电子科技大学 | Combined estimator image defogging method based on convolutional neural networks |
| CN109146810A (en) * | 2018-08-08 | 2019-01-04 | 国网浙江省电力有限公司信息通信分公司 | A kind of image defogging method based on end-to-end deep learning |
Non-Patent Citations (2)
| Title |
|---|
| 二阶段端到端的图像去雾生成网络;邢晓敏等;《计算机辅助设计与图形学学报》;20200115(第01期);全文 * |
| 基于对偶学习的图像去雾网络;丛晓峰等;《应用光学》;20200115(第01期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111553856A (en) | 2020-08-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111553856B (en) | Image defogging method based on depth estimation assistance | |
| CN107123089B (en) | Remote sensing image super-resolution reconstruction method and system based on depth convolution network | |
| Yang et al. | Proximal dehaze-net: A prior learning-based deep network for single image dehazing | |
| CN114092330B (en) | Light-weight multi-scale infrared image super-resolution reconstruction method | |
| CN111915530B (en) | End-to-end-based haze concentration self-adaptive neural network image defogging method | |
| CN108269244B (en) | Image defogging system based on deep learning and prior constraint | |
| CN110517203B (en) | Defogging method based on reference image reconstruction | |
| CN110544213B (en) | Image defogging method based on global and local feature fusion | |
| CN113066028B (en) | Image defogging method based on transform deep neural network | |
| CN109584170B (en) | Underwater image restoration method based on convolutional neural network | |
| CN111161360B (en) | Image defogging method of end-to-end network based on Retinex theory | |
| CN110443761B (en) | Single image rain removing method based on multi-scale aggregation characteristics | |
| CN111784602A (en) | Method for generating countermeasure network for image restoration | |
| CN110349087B (en) | RGB-D image high-quality grid generation method based on adaptive convolution | |
| CN110223251B (en) | Convolution neural network underwater image restoration method suitable for artificial and natural light sources | |
| CN113284070A (en) | Non-uniform fog image defogging algorithm based on attention transfer mechanism | |
| CN116051396B (en) | Image denoising method based on feature enhancement network and GRU network | |
| CN114820408A (en) | Infrared and visible light image fusion method based on self-attention and convolutional neural network | |
| Fan et al. | Multiscale cross-connected dehazing network with scene depth fusion | |
| CN111667421A (en) | Image defogging method | |
| CN114581304A (en) | Image super-resolution and defogging fusion method and system based on circulating network | |
| CN113160286A (en) | Near-infrared and visible light image fusion method based on convolutional neural network | |
| Wang et al. | Multiscale supervision-guided context aggregation network for single image dehazing | |
| CN112070688A (en) | Single image defogging method for generating countermeasure network based on context guidance | |
| CN115222614A (en) | Priori-guided multi-degradation-characteristic night light remote sensing image quality improving method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |