CN110097522B - Single outdoor image defogging method based on multi-scale convolution neural network - Google Patents

Single outdoor image defogging method based on multi-scale convolution neural network Download PDF

Info

Publication number
CN110097522B
CN110097522B CN201910397724.7A CN201910397724A CN110097522B CN 110097522 B CN110097522 B CN 110097522B CN 201910397724 A CN201910397724 A CN 201910397724A CN 110097522 B CN110097522 B CN 110097522B
Authority
CN
China
Prior art keywords
neural network
scale
layer
convolution
convolutional neural
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910397724.7A
Other languages
Chinese (zh)
Other versions
CN110097522A (en
Inventor
张世辉
桑榆
陈宇翔
张健
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yanshan University
Beijing Institute of Computer Technology and Applications
Original Assignee
Yanshan University
Beijing Institute of Computer Technology and Applications
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yanshan University, Beijing Institute of Computer Technology and Applications filed Critical Yanshan University
Priority to CN201910397724.7A priority Critical patent/CN110097522B/en
Publication of CN110097522A publication Critical patent/CN110097522A/en
Application granted granted Critical
Publication of CN110097522B publication Critical patent/CN110097522B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20024—Filtering details
    • G06T2207/20028—Bilateral filtering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20192—Edge enhancement; Edge preservation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a single outdoor image defogging method based on a multi-scale convolutional neural network, and belongs to the field of computer vision. The invention comprises the following steps: constructing a training sample set according to the atmospheric scattering model; building a multi-scale convolutional neural network based on the deep learning idea; constructing a target function according to the built multi-scale convolutional neural network; and training the multi-scale convolutional neural network based on the constructed objective function. The invention does not need to acquire the prior knowledge of the outdoor image and can effectively store the information of the edge, the texture, the color, the contrast, the saturation and the like of the image.

Description

Single outdoor image defogging method based on multi-scale convolution neural network
Technical Field
The invention belongs to the field of computer vision, and particularly relates to a single outdoor image defogging method based on a multi-scale convolutional neural network.
Background
Fog is a traditional atmospheric phenomenon formed by particles of water vapor, dust, smoke, and the like. Fog can cause blurring, contrast reduction, saturation deviation of images processed by a vision system, further hinder the performance of visual tasks such as classification, identification, detection and tracking, and even cause failure of related visual tasks. Therefore, how to remove fog from outdoor images becomes a difficult problem in the field of computer vision and is receiving wide attention from scholars.
There are two main types of information handled by existing defogging methods: outdoor video and a single outdoor image. The defogging method based on the outdoor video is relatively few, and the main reason is that the defogging method based on the video firstly needs to divide the video into a plurality of video frames and then sequentially defogges the divided video frames, and essentially still processes a single outdoor image. Therefore, existing defogging methods are generally implemented based on a single outdoor image. Meanwhile, the existing defogging method based on the single outdoor image has the problems of prior knowledge acquisition, edge and texture loss, color, contrast and saturation distortion and the like. The defogging method based on dark channel prior proposed by K.M.He and J.Sun in the article "Single image size removal using dark channel prior. proceedings of the IEEE Conference on Computer Vision and Pattern registration works: IEEE Computer Society,2009: 1956-. The defogging method based on the improved dark channel prior, which is proposed in the article of Chengzhou Zhen and Zhanzhu Guangzhou, "Single image defogging algorithm based on the improved dark channel prior and guided filtering," automatic chemistry report, 2016,42(3):455-465, "has the problems that the dark channel threshold and the maximum value of the mixed dark channel brightness cannot be selected in a self-adaptive manner, and the color of the defogged image is distorted. The methods proposed by C.Z.He and C.D.Zhang in the article "A size dense adaptive texture size removal algorithm. proceedings of the IEEE International Conference on Information and Automation, IEEE Computer Society,2016: 1933-. The methods proposed by B Cai and X Xu in the article "DehazeNet: An End-to-End System for Single Image Haze removal. IEEE Transactions on Image Processing,2016,25(11): 5187-. The methods proposed by z.g.ling and g.f.fan in the article "performance oriented transmission estimation for high quality image smoothing. neuro-compressing, 2017,224(2): 82-95" have problems with distortion of contrast and saturation. The outdoor images, defogged by the methods proposed by Z.G.Li and H.J. in the article "Single Image De-Hazing Using Global Guided Image Filter, IEEE Transactions on Image Processing,2018,27(1): 442-450", are color distorted and have low contrast.
Disclosure of Invention
Aiming at the problems of the existing single outdoor image defogging method, the invention aims to provide the single outdoor image defogging method based on the multi-scale convolutional neural network.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows: a single outdoor image defogging method based on a multi-scale convolution neural network is characterized by comprising the following steps:
(1) obtaining a training sample: acquiring a fog-free image sample, carrying out atomization treatment on the fog-free image sample by using an atmospheric scattering model to obtain a fog image sample, and taking the fog-free image sample and the corresponding fog image sample as training samples;
(2) multi-scale convolutional neural network model: constructing at least three convolution layers in parallel; the output end of each convolution layer is connected with a Max Pooling Pooling layer, and the output end of each Max Pooling Pooling layer is connected with a nonlinear mapping layer based on a modified linear unit ReLu; the output ends of all the nonlinear mapping layers are connected with the characteristic fusion layer; the output end of the characteristic fusion layer is connected with a bilateral filter layer for processing the transmissivity, and the transmissivity output by the bilateral filter layer is used for defogging the foggy image sample input by the convolution layer;
(3) training a multi-scale convolution neural network model: training the multi-scale convolutional neural network model by using the foggy image sample in the step (1) as the input of the multi-scale convolutional neural network model and using the fogless image sample in the step (1) as the discrimination standard of the output of the multi-scale neural network, and aiming at the minimization of an objective function, and performing parameter solution; wherein the objective function is:
Figure GDA0002779361230000031
wherein, ci、siAnd hiThe average RGB value, the average contrast and the average saturation corresponding to the ith sample are respectively; the parameter of the multi-scale convolution neural network is phi, and the ith hazy image sample is IiAnd the fog-free image sample corresponding to the ith fog image sample is JiThe number of training samples is N;
(4) and (4) carrying out defogging treatment on the foggy image to be treated by utilizing the multi-scale convolution neural network model solved in the step (3).
The further technical scheme is that the number of the convolution layers is three, and the convolution kernels are respectively 7 × 7,5 × 5 and 3 × 3; or four convolution layers, the convolution kernels being 11 × 11,7 × 7,5 × 5 and 3 × 3, respectively.
The further technical scheme is that the objective function is constructed according to Mean Square Error (MSE) and L-2 norm.
The further technical scheme is that the objective function is minimized according to a random gradient descent method.
The further technical scheme is that in the step (1) of obtaining the sample, the calculation method of the fog-free image sample structure corresponding to the fog image sample is as follows:
IT(x)＝JT(x)tT(x)+αT(1-tT(x))
wherein, JT(x) Is a fog-free image sample, i.e. the image collected, tT(x) Is a transmittance, αTIs a global atmospheric light value, IT(x) Is a hazy image sample.
The further technical proposal is that the form of the convolution layer is as follows:
Figure GDA0002779361230000041
where I (x) is the hazy image sample to be dehazed, q is the convolution kernel size,
Figure GDA0002779361230000042
is a convolutional layer filterThe wave filter is used for filtering the received signal,
Figure GDA0002779361230000043
is convolution layer bias, is convolution operation,
Figure GDA0002779361230000044
is the output of the convolution layer of the multi-scale convolution neural network.
The further technical scheme is that the Max Pooling layer is shown as follows:
Figure GDA0002779361230000045
wherein the content of the first and second substances,
Figure GDA0002779361230000046
is the output of the multi-scale convolutional neural network pooling layer,
Figure GDA0002779361230000047
is the output of the convolution layer of the multi-scale convolution neural network.
The further technical scheme is that the nonlinear mapping layer performs nonlinear mapping on the reduced-dimension features to obtain a multi-scale feature map, and the form of the constructed activation layer is as follows:
Figure GDA0002779361230000051
wherein the content of the first and second substances,
Figure GDA0002779361230000052
is an active-layer filter that is,
Figure GDA0002779361230000053
it is the bias of the active layer that,
Figure GDA0002779361230000054
is the output of the multi-scale convolutional neural network activation layer,
Figure GDA0002779361230000055
is the output of the multi-scale convolutional neural network pooling layer.
The further technical scheme is that the characteristic fusion layer fuses the multi-scale characteristic graphs so as to obtain the transmissivity
Figure GDA0002779361230000058
The multi-scale feature map fusion mode is as follows:
Figure GDA0002779361230000056
wherein λ is1,λ2,…,λnAre respectively the feature map weight coefficients, h(q)、c(q)And s(q)Respectively obtaining an average RGB value, an average contrast value and an average saturation value of each scale feature map;
Figure GDA0002779361230000057
is the output of the pooling layer of the n-scale convolutional neural network.
The technical scheme is that the bilateral filter layer utilizes bilateral filter pair transmissivity
Figure GDA0002779361230000059
Processing is carried out, so that refined transmittance t (x) is obtained, and the calculation method is as follows:
Figure GDA0002779361230000061
Figure GDA0002779361230000062
d(ξ,y)＝||ξ-y||2
Figure GDA0002779361230000063
Figure GDA0002779361230000064
wherein y is the transmittance
Figure GDA0002779361230000065
Where xi is the pixel adjacent to y, c (xi, y) is the space weight function, and σ iscIs the variance between two pixels, d (ξ, y) is the distance metric between two pixels,
Figure GDA0002779361230000066
and
Figure GDA0002779361230000067
respectively, the transmissivity formed by 8 neighborhood pixel blocks taking xi and y as centers,
Figure GDA0002779361230000068
is a function of similarity weight calculation, σsIs the variance between the two transmittances and,
Figure GDA0002779361230000069
is a function of two transmittance distance measures.
Compared with the prior art, the invention has the advantages that:
(1) based on the deep learning idea, a multi-scale convolutional neural network is built, and the characteristics of feature maps of all scales are fully excavated, so that the information of the defogged outdoor image, such as color, contrast, saturation and the like, is similar to the initial foggy outdoor image.
(2) And processing the transmissivity obtained by fusing the characteristic graphs by utilizing bilateral filtering, so that the edge and texture information of the defogged outdoor image is completely stored.
(3) And constructing an objective function based on MSE and L-2 norm, thereby realizing effective fitting of the multi-scale convolutional neural network and more effective removal of fog in the outdoor image.
Drawings
FIG. 1 is a flow chart of a defogging method according to the present invention;
FIG. 2 is a schematic diagram of a portion of a training sample;
FIG. 3 is a schematic diagram of a 3-scale convolutional neural network structure;
fig. 4 is a schematic diagram of a 4-scale convolutional neural network structure.
Detailed Description
In order to make the technical scheme of the present invention clearer, the present invention is further explained with reference to the accompanying drawings.
The embodiment of the invention discloses a single outdoor image defogging method based on a multi-scale convolution neural network, which is characterized by comprising the following steps of:
(1) obtaining a training sample: acquiring a fog-free image sample, carrying out atomization treatment on the fog-free image sample by using an atmospheric scattering model to obtain a fog image sample, and taking the fog-free image sample and the corresponding fog image sample as training samples;
(2) multi-scale convolutional neural network model: constructing at least three convolution layers in parallel; the output end of each convolution layer is connected with a Max Pooling Pooling layer, and the output end of each Max Pooling Pooling layer is connected with a nonlinear mapping layer based on a modified linear unit ReLu; the output ends of all the nonlinear mapping layers are connected with the characteristic fusion layer; the output end of the characteristic fusion layer is connected with a bilateral filter layer for processing the transmissivity, and the transmissivity output by the bilateral filter layer is used for defogging the foggy image sample input by the convolution layer;
(3) training a multi-scale convolution neural network model: training the multi-scale convolutional neural network model by using the foggy image sample in the step (1) as the input of the multi-scale convolutional neural network model and using the fogless image sample in the step (1) as the discrimination standard of the output of the multi-scale neural network, and aiming at the minimization of an objective function, and performing parameter solution; wherein the objective function is:
Figure GDA0002779361230000081
wherein, ci、siAnd hiThe average RGB value, the average contrast and the average saturation corresponding to the ith sample are respectively; the parameter of the multi-scale convolution neural network is phi, and the ith hazy image sample is IiAnd the fog-free image sample corresponding to the ith fog image sample is JiThe number of training samples is N;
(4) and (4) carrying out defogging treatment on the foggy image to be treated by utilizing the multi-scale convolution neural network model solved in the step (3).
In the embodiment of the invention, the number of the convolution layers is three, and the convolution kernels are respectively 7 × 7,5 × 5 and 3 × 3; or four convolution layers, the convolution kernels being 11 × 11,7 × 7,5 × 5 and 3 × 3, respectively.
The objective function in the embodiment of the invention is constructed according to Mean Square Error (MSE) and L-2 norm.
The objective function in the embodiment of the invention is minimized according to a random gradient descent method.
In the embodiment of the invention, in the step (1) of obtaining the sample, a method for calculating a fog image sample corresponding to a fog-free image sample structure is as follows:
IT(x)＝JT(x)tT(x)+αT(1-tT(x))
wherein, JT(x) Is a fog-free image sample, i.e. the image collected, tT(x) Is a transmittance, αTIs a global atmospheric light value, IT(x) Is a hazy image sample.
The form of the convolutional layer in the embodiment of the present invention is as follows:
Figure GDA0002779361230000082
where I (x) is the hazy image sample to be dehazed, q is the convolution kernel size,
Figure GDA0002779361230000083
is a convolutional layer filter which is a convolutional layer filter,
Figure GDA0002779361230000091
is convolution layer bias, is convolution operation,
Figure GDA0002779361230000092
is the output of the convolution layer of the multi-scale convolution neural network.
The Max Pooling layer in the embodiment of the present invention is shown as follows:
Figure GDA0002779361230000093
wherein the content of the first and second substances,
Figure GDA0002779361230000094
is the output of the multi-scale convolutional neural network pooling layer,
Figure GDA0002779361230000095
is the output of the convolution layer of the multi-scale convolution neural network.
In the embodiment of the invention, the nonlinear mapping layer performs nonlinear mapping on the reduced-dimension features to obtain a multi-scale feature map, and the form of the constructed activation layer is as follows:
Figure GDA0002779361230000096
wherein the content of the first and second substances,
Figure GDA0002779361230000097
is an active-layer filter that is,
Figure GDA0002779361230000098
it is the bias of the active layer that,
Figure GDA0002779361230000099
is the output of the multi-scale convolutional neural network activation layer,
Figure GDA00027793612300000910
is the output of the multi-scale convolutional neural network pooling layer.
The characteristic fusion layer in the embodiment of the invention fuses the multi-scale characteristic graphs to obtain the transmissivity
Figure GDA00027793612300000913
The multi-scale feature map fusion mode is as follows:
Figure GDA00027793612300000911
wherein λ is1,λ2,…,λnAre respectively the feature map weight coefficients, h(q)、c(q)And s(q)Respectively obtaining an average RGB value, an average contrast value and an average saturation value of each scale feature map;
Figure GDA00027793612300000912
is the output of the pooling layer of the n-scale convolutional neural network.
The bilateral filter layer in the embodiment of the invention utilizes bilateral filter pair transmissivity
Figure GDA0002779361230000101
Processing is carried out, so that refined transmittance t (x) is obtained, and the calculation method is as follows:
Figure GDA0002779361230000102
Figure GDA0002779361230000103
d(ξ,y)＝||ξ-y||2
Figure GDA0002779361230000104
Figure GDA0002779361230000105
wherein y is the transmittance
Figure GDA0002779361230000106
Where xi is the pixel adjacent to y, c (xi, y) is the space weight function, and σ iscIs the variance between two pixels, d (ξ, y) is the distance metric between two pixels,
Figure GDA0002779361230000107
and
Figure GDA0002779361230000108
respectively, the transmissivity formed by 8 neighborhood pixel blocks taking xi and y as centers,
Figure GDA0002779361230000109
is a function of similarity weight calculation, σsIs the variance between the two transmittances and,
Figure GDA00027793612300001010
is a function of two transmittance distance measures.
In the embodiment of the present invention, as shown in fig. 1, the single outdoor image defogging method based on the multi-scale convolutional neural network includes the following steps:
step 1: and obtaining a training sample according to the atmospheric scattering model, and constructing a training sample data set.
1.1) collecting 3000 fog-free outdoor images under different scenes from the Internet.
1.2) for 3000 outdoor images collected, and setting JT(x) Is a fog-free outdoor image, i.e. an image collected, tT(x) Is a transmittance, αTIs a global atmospheric light value, IT(x) Is a foggy outdoor image. Different t is selected to ensure that the training sample contains multiple conditions as much as possibleT(x) And global atmospheric light value alphaTSelecting a fixed value, and defining a foggy outdoor image IT(x) Is composed of
IT(x)＝JT(x)tT(x)+αT(1-tT(x)) (1)
1.3, traversing 3000 outdoor images, and acquiring 3000 foggy outdoor images as training samples according to the above formula, thereby constructing a training sample data set.
Some training samples in the training sample data set and their group Truth are shown in FIG. 2. The first column is a fog-free outdoor image acquired from the internet, and the second column is a fog-containing outdoor image calculated by the formula (1).
Step 2: and constructing a multi-scale convolutional neural network.
2.1) the built convolution layer of the multi-scale convolution neural network consists of convolution kernels with three different scales of 7 x 7,5 x 5 and 3 x 3, a training sample to be defogged is set as I (x), q represents the size of the convolution kernel and belongs to {7,5 and 3},
Figure GDA0002779361230000111
which represents a convolutional layer filter, is,
Figure GDA0002779361230000112
representing convolutional layer bias, representing convolution operation, and outputting convolutional layer of multi-scale convolutional neural network
Figure GDA0002779361230000113
Can be expressed as
Figure GDA0002779361230000114
2.2) the Pooling layer of the multi-scale neural network built by Max Pooling is constructed, aiming at reducing the dimension of the calculated characteristics after the convolution layer, thereby obtaining the characteristics with translation invariance and rotation invariance, and then the output of the Pooling layer of the multi-scale convolutional neural network
Figure GDA0002779361230000115
Can be expressed as
Figure GDA0002779361230000116
2.3) constructing an activation layer by the built multi-scale convolution neural network according to the modified linear unit ReLu, carrying out nonlinear mapping on the reduced-dimension characteristics to obtain a multi-scale characteristic diagram,
Figure GDA0002779361230000117
which represents the filter of the active layer,
Figure GDA0002779361230000118
representing the bias of the active layer, the output of the active layer of the multi-scale convolutional neural network
Figure GDA0002779361230000119
Can be expressed as
Figure GDA0002779361230000121
2.4) after obtaining the multi-scale characteristic diagram, fully excavating the characteristics of the color, the saturation and the contrast of each scale characteristic diagram, and fusing each scale characteristic diagram, thereby obtaining the transmissivity corresponding to the input image I (x) by calculation
Figure GDA0002779361230000123
The fusion function is defined as
Figure GDA0002779361230000122
Wherein, λ, μ and γ are feature map weight coefficients, h(q)、c(q)And s(q)The average RGB value, the average contrast value and the average saturation value of each scale feature map are respectively.
2.5) because the transmissivity calculated by the existing defogging method is rough, the defogged outdoor image generally has incomplete boundary and texture preservation and the likeAnd (5) problems are solved. Therefore, bilateral filtering is selected to process the transmittance obtained by feature map fusion to obtain refined transmittance t (x), and the transmittance is processed
Figure GDA0002779361230000124
Performing bilateral filtering process may be expressed as
Figure GDA0002779361230000131
Wherein y is the transmittance
Figure GDA0002779361230000132
Where xi is the pixel adjacent to y, c (xi, y) is the space weight function, and σ iscIs the variance between two pixels, d (ξ, y) is the distance metric between two pixels,
Figure GDA0002779361230000133
and
Figure GDA0002779361230000135
respectively, the transmissivity formed by 8 neighborhood pixel blocks taking xi and y as centers,
Figure GDA0002779361230000136
is a function of similarity weight calculation, σsIs the variance between the two transmittances and,
Figure GDA0002779361230000137
is a function of two transmittance distance measures.
2.6) after obtaining the refined transmittance t (x), selecting the global atmospheric light value alpha as the maximum brightness value corresponding to each pixel point in the input image I (x). At the moment, the atmosphere scattering model is deformed, so that the outdoor image J (x) after defogging is obtained through calculation, and the calculation method is that
Figure GDA0002779361230000138
The multi-scale convolutional neural network structure is shown in fig. 3.
And step 3: and constructing an objective function according to the mean square error MSE and the L-2 norm.
The single outdoor image defogging problem is a typical supervised learning problem, and the supervised learning needs to establish a mapping relation G between the input (foggy outdoor image) and the output (fogless outdoor image) of a convolutional neural network. Setting parameters of the multi-scale convolutional neural network as
Figure GDA0002779361230000139
The ith training sample is IiThe group Truth corresponding to the ith training sample is JiThe number of training samples is N, the parameter phi of the multi-scale convolutional neural network can be obtained by minimizing an objective function, and the objective function is constructed by mean square error MSE and L-2 norm and has the form of
Figure GDA0002779361230000141
Wherein, ci、siAnd hiThe average RGB value, the average contrast and the average saturation corresponding to the ith sample are respectively.
And 4, step 4: and training the multi-scale convolutional neural network.
Firstly, 20000 64 multiplied by 64 foggy image blocks are randomly extracted from a constructed training sample set, and each foggy image block has a corresponding Ground Truth; then, minimizing the constructed objective function by utilizing a random gradient descent method; finally, setting a threshold value for the target function, and when the result of the minimized target function is smaller than the set threshold value, namely the parameter phi representing the multi-scale convolutional neural network is determined, finishing the training of the convolutional neural network at the moment, and further realizing the defogging treatment on any outdoor image; as shown in fig. 3, the output of the multi-scale convolutional neural network is a processed picture.
The embodiment of the invention improves the previous embodiment, wherein in the improvement, the convolutional layer for constructing the multi-scale convolutional neural network in the step 2 is composed of convolution kernels with four different scales of 11 × 11,7 × 7,5 × 5 and 3 × 3, a training sample to be defogged is set as I (x), q represents the size of the convolution kernel and belongs to the {11,7,5 and 3}, and the fusion mode of the multi-scale feature map is as follows:
Figure GDA0002779361230000151
wherein, λ, μ, γ and β are characteristic map weight coefficients, h(q)、c(q)And s(q)The average RGB value, the average contrast value and the average saturation value of each scale feature map are respectively. As shown in fig. 4, the output of the four-scale convolutional neural network is a processed picture.
The above-mentioned embodiments are merely illustrative of the preferred embodiments of the present invention, and do not limit the scope of the present invention, and various modifications and improvements made to the technical solution of the present invention by those skilled in the art without departing from the spirit of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims (10)

1. A single outdoor image defogging method based on a multi-scale convolution neural network is characterized by comprising the following steps:
(1) obtaining a training sample: acquiring a fog-free image sample, carrying out atomization treatment on the fog-free image sample by using an atmospheric scattering model to obtain a fog image sample, and taking the fog-free image sample and the corresponding fog image sample as training samples;
(2) multi-scale convolutional neural network model: constructing at least three convolution layers in parallel; the output end of each convolution layer is connected with a Max Pooling Pooling layer, and the output end of each Max Pooling Pooling layer is connected with a nonlinear mapping layer based on a modified linear unit ReLu; the output ends of all the nonlinear mapping layers are connected with the characteristic fusion layer; the output end of the characteristic fusion layer is connected with a bilateral filter layer for processing the transmissivity, and the transmissivity output by the bilateral filter layer is used for defogging the foggy image sample input by the convolution layer;
(3) training a multi-scale convolution neural network model: training the multi-scale convolutional neural network model by using the foggy image sample in the step (1) as the input of the multi-scale convolutional neural network model and using the fogless image sample in the step (1) as the discrimination standard of the output of the multi-scale neural network, and aiming at the minimization of an objective function, and performing parameter solution; wherein the objective function is:
Figure FDA0002779361220000011
wherein, ci、siAnd hiThe average RGB value, the average contrast and the average saturation corresponding to the ith sample are respectively; the parameter of the multi-scale convolution neural network is phi, and the ith hazy image sample is IiAnd the fog-free image sample corresponding to the ith fog image sample is JiThe number of training samples is N;
(4) and (4) carrying out defogging treatment on the foggy image to be treated by utilizing the multi-scale convolution neural network model solved in the step (3).
2. The single outdoor image defogging method based on the multi-scale convolutional neural network as claimed in claim 1, wherein the number of the convolutional layers is three, and the convolutional kernels are respectively 7 x 7,5 x 5 and 3 x 3; or four convolution layers, the convolution kernels being 11 × 11,7 × 7,5 × 5 and 3 × 3, respectively.
3. The single outdoor image defogging method based on the multi-scale convolutional neural network as claimed in claim 1, wherein the objective function is constructed according to Mean Square Error (MSE) and L-2 norm.
4. The method of claim 1, wherein the objective function is minimized according to a stochastic gradient descent method.
5. The single outdoor image defogging method based on the multi-scale convolutional neural network as claimed in claim 1, wherein in the step (1) of acquiring the samples, the calculation method of the fog-free image sample structure corresponding to the fog image sample is as follows:
IT(x)＝JT(x)tT(x)+αT(1-tT(x))
wherein, JT(x) Is a fog-free image sample, i.e. the image collected, tT(x) Is a transmittance, αTIs a global atmospheric light value, IT(x) Is a hazy image sample.
6. The single outdoor image defogging method based on the multi-scale convolutional neural network as claimed in claim 1 or 2, wherein the convolutional layer form is as follows:
Figure FDA0002779361220000021
where I (x) is the hazy image sample to be dehazed, q is the convolution kernel size,
Figure FDA0002779361220000022
is a convolutional layer filter which is a convolutional layer filter,
Figure FDA0002779361220000023
is convolution layer bias, is convolution operation,
Figure FDA0002779361220000024
is the output of the convolution layer of the multi-scale convolution neural network.
7. The single outdoor image defogging method based on the multi-scale convolutional neural network as claimed in claim 1, wherein the Max Pooling Pooling layer form is as follows:
Figure FDA0002779361220000031
wherein the content of the first and second substances,
Figure FDA0002779361220000032
is the output of the multi-scale convolutional neural network pooling layer,
Figure FDA0002779361220000033
is the output of the convolution layer of the multi-scale convolution neural network.
8. The single outdoor image defogging method based on the multi-scale convolutional neural network as claimed in claim 1, wherein the nonlinear mapping layer is used for carrying out nonlinear mapping on the dimensionality reduced features to obtain a multi-scale feature map, and the constructed activation layer is in the following form:
Figure FDA0002779361220000034
wherein the content of the first and second substances,
Figure FDA0002779361220000035
is an active-layer filter that is,
Figure FDA0002779361220000036
it is the bias of the active layer that,
Figure FDA0002779361220000037
is the output of the multi-scale convolutional neural network activation layer,
Figure FDA0002779361220000038
is the output of the multi-scale convolutional neural network pooling layer.
9. The single outdoor image defogger based on the multi-scale convolutional neural network as claimed in claim 1A method wherein the feature fusion layer fuses multi-scale feature maps to obtain the transmittance
Figure FDA0002779361220000039
The multi-scale feature map fusion mode is as follows:
Figure FDA00027793612200000310
wherein λ is1,λ2,…,λnAre respectively the feature map weight coefficients, h(q)、c(q)And s(q)Respectively obtaining an average RGB value, an average contrast value and an average saturation value of each scale feature map;
Figure FDA0002779361220000041
is the output of the pooling layer of the n-scale convolutional neural network.
10. The method as claimed in claim 1, wherein the bilateral filter layer utilizes bilateral filtering to transmit power
Figure FDA0002779361220000042
Processing is carried out, so that refined transmittance t (x) is obtained, and the calculation method is as follows:
Figure FDA0002779361220000043
Figure FDA0002779361220000044
Figure FDA0002779361220000045
Figure FDA0002779361220000046
Figure FDA0002779361220000047
wherein y is the transmittance
Figure FDA0002779361220000048
Where xi is the pixel adjacent to y, c (xi, y) is the space weight function, and σ iscIs the variance between two pixels, d (ξ, y) is the distance metric between two pixels,
Figure FDA0002779361220000049
and
Figure FDA00027793612200000410
respectively, the transmissivity formed by 8 neighborhood pixel blocks taking xi and y as centers,
Figure FDA00027793612200000411
is a function of similarity weight calculation, σsIs the variance between the two transmittances and,
Figure FDA00027793612200000412
is a function of two transmittance distance measures.
CN201910397724.7A 2019-05-14 2019-05-14 Single outdoor image defogging method based on multi-scale convolution neural network Active CN110097522B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910397724.7A CN110097522B (en) 2019-05-14 2019-05-14 Single outdoor image defogging method based on multi-scale convolution neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910397724.7A CN110097522B (en) 2019-05-14 2019-05-14 Single outdoor image defogging method based on multi-scale convolution neural network

Publications (2)

Publication Number Publication Date
CN110097522A CN110097522A (en) 2019-08-06
CN110097522B true CN110097522B (en) 2021-03-19

Family

ID=67447928

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910397724.7A Active CN110097522B (en) 2019-05-14 2019-05-14 Single outdoor image defogging method based on multi-scale convolution neural network

Country Status (1)

Country Link
CN (1) CN110097522B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110570371B (en) * 2019-08-28 2023-08-29 天津大学 Image defogging method based on multi-scale residual error learning
CN110738622A (en) * 2019-10-17 2020-01-31 温州大学 Lightweight neural network single image defogging method based on multi-scale convolution
CN111369472B (en) * 2020-03-12 2021-04-23 北京字节跳动网络技术有限公司 Image defogging method and device, electronic equipment and medium
CN112164010A (en) * 2020-09-30 2021-01-01 南京信息工程大学 Multi-scale fusion convolution neural network image defogging method
CN114049274A (en) * 2021-11-13 2022-02-15 哈尔滨理工大学 Defogging method for single image

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107749052A (en) * 2017-10-24 2018-03-02 中国科学院长春光学精密机械与物理研究所 Image defogging method and system based on deep learning neutral net
CN108269244A (en) * 2018-01-24 2018-07-10 东北大学 It is a kind of based on deep learning and prior-constrained image defogging system
CN109360156A (en) * 2018-08-17 2019-02-19 上海交通大学 Single image rain removing method based on the image block for generating confrontation network

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106960425A (en) * 2017-04-05 2017-07-18 上海矽奥微电子有限公司 Single frames defogging method based on multiple dimensioned filtering of deconvoluting
JP7146372B2 (en) * 2017-06-21 2022-10-04 キヤノン株式会社 Image processing device, imaging device, image processing method, program, and storage medium
CN108564549B (en) * 2018-04-20 2022-04-05 福建帝视信息科技有限公司 Image defogging method based on multi-scale dense connection network
CN109087254B (en) * 2018-04-26 2021-12-31 长安大学 Unmanned aerial vehicle aerial image haze sky and white area self-adaptive processing method
CN109360155B (en) * 2018-08-17 2020-10-13 上海交通大学 Single-frame image rain removing method based on multi-scale feature fusion
CN109410144B (en) * 2018-10-31 2020-11-27 聚时科技(上海)有限公司 End-to-end image defogging processing method based on deep learning
CN109712083B (en) * 2018-12-06 2021-02-12 南京邮电大学 Single image defogging method based on convolutional neural network
CN109584188B (en) * 2019-01-15 2022-11-11 东北大学 Image defogging method based on convolutional neural network

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107749052A (en) * 2017-10-24 2018-03-02 中国科学院长春光学精密机械与物理研究所 Image defogging method and system based on deep learning neutral net
CN108269244A (en) * 2018-01-24 2018-07-10 东北大学 It is a kind of based on deep learning and prior-constrained image defogging system
CN109360156A (en) * 2018-08-17 2019-02-19 上海交通大学 Single image rain removing method based on the image block for generating confrontation network

Also Published As

Publication number Publication date
CN110097522A (en) 2019-08-06

Similar Documents

Publication Publication Date Title
CN110097522B (en) Single outdoor image defogging method based on multi-scale convolution neural network
CN110570371B (en) Image defogging method based on multi-scale residual error learning
CN109272455B (en) Image defogging method based on weak supervision generation countermeasure network
CN107301624B (en) Convolutional neural network defogging method based on region division and dense fog pretreatment
Pan et al. Underwater image de-scattering and enhancing using dehazenet and HWD
CN108510451B (en) Method for reconstructing license plate based on double-layer convolutional neural network
CN111161360B (en) Image defogging method of end-to-end network based on Retinex theory
CN110675340A (en) Single image defogging method and medium based on improved non-local prior
Zhou et al. FSAD-Net: feedback spatial attention dehazing network
CN111582074A (en) Monitoring video leaf occlusion detection method based on scene depth information perception
CN115330623A (en) Image defogging model construction method and system based on generation countermeasure network
Bansal et al. A review of image restoration based image defogging algorithms
CN114219732A (en) Image defogging method and system based on sky region segmentation and transmissivity refinement
CN112950589A (en) Dark channel prior defogging algorithm of multi-scale convolution neural network
CN112164010A (en) Multi-scale fusion convolution neural network image defogging method
Fu et al. An anisotropic Gaussian filtering model for image de-hazing
Chen et al. Improve transmission by designing filters for image dehazing
Wang et al. Single-image dehazing using color attenuation prior based on haze-lines
Hao et al. Texture enhanced underwater image restoration via laplacian regularization
CN110738624A (en) area self-adaptive image defogging system and method
Thepade et al. Improved haze removal method using proportionate fusion of color attenuation prior and edge preserving
CN112184566B (en) Image processing method and system for removing adhered water mist and water drops
CN111932470A (en) Image restoration method, device, equipment and medium based on visual selection fusion
CN111932469A (en) Significance weight quick exposure image fusion method, device, equipment and medium
Shivakumar et al. Remote sensing and natural image dehazing using DCP based IDERS framework

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant