CN111507909A - Method and device for clearing fog image and storage medium - Google Patents
Method and device for clearing fog image and storage medium Download PDFInfo
- Publication number
- CN111507909A CN111507909A CN202010193972.2A CN202010193972A CN111507909A CN 111507909 A CN111507909 A CN 111507909A CN 202010193972 A CN202010193972 A CN 202010193972A CN 111507909 A CN111507909 A CN 111507909A
- Authority
- CN
- China
- Prior art keywords
- image
- fog
- generator
- network
- free
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000012549 training Methods 0.000 claims abstract description 37
- 230000008447 perception Effects 0.000 claims abstract description 33
- 125000004122 cyclic group Chemical group 0.000 claims abstract description 28
- 230000003042 antagnostic effect Effects 0.000 claims abstract description 17
- 230000009467 reduction Effects 0.000 claims abstract description 9
- 238000003707 image sharpening Methods 0.000 claims description 25
- 238000004590 computer program Methods 0.000 claims description 24
- 230000006870 function Effects 0.000 claims description 24
- 238000012360 testing method Methods 0.000 claims description 11
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 9
- 238000010276 construction Methods 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 4
- 230000006735 deficit Effects 0.000 claims description 3
- 238000005352 clarification Methods 0.000 claims 2
- 230000000306 recurrent effect Effects 0.000 claims 1
- 238000006243 chemical reaction Methods 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 7
- 230000000694 effects Effects 0.000 description 6
- 230000006872 improvement Effects 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 235000019646 color tone Nutrition 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011158 quantitative evaluation Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a method, a device and a storage medium for clearing a foggy image, wherein the method comprises the following steps: acquiring a foggy image dataset and a fogless image dataset; constructing a loop to generate an antagonistic network and a perceptual loss network; the loop generation countermeasure network comprises a first generator, a second generator, a first discriminator and a second discriminator; inputting the foggy image data set and the fogless image data set into a cyclic generation countermeasure network, and training by combining the perception loss network to obtain an optimal generation model; inputting the image to be defogged into the optimal generation model to obtain a corresponding predicted fog-free image; and performing Laplacian pyramid reduction on the predicted fog-free image to obtain a clear fog-free image. The method can automatically extract the characteristics of the foggy image, complete the style conversion of the single foggy image and the fogless image, and does not need to acquire the foggy image and the real fogless image which are matched in pairs under the same scene for training, thereby conveniently and flexibly making the foggy image clear.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method and an apparatus for sharpening a foggy image, and a storage medium.
Background
The single fog image is cleared by adopting a series of methods to remove the interference of fog in the image, thereby recovering a high-definition image.
Conventional fogging image sharpening methods fall into two main categories: the single image defogging based on the prior condition refers to parameter estimation of an atmospheric scattering model by using prior information, and because the assumption of the prior condition is not always true in a specific scene, the defogged image cannot be well cleared in some cases by the prior method. Most of CNN fog image sharpening models based on deep learning need to evaluate intermediate parameters of atmosphere scattering models, and need to input paired fog images and corresponding ground real images, but it is also difficult to simultaneously acquire paired fog images and real fog-free images in the same scene. Therefore, the traditional method for clearing the foggy image has certain limitations.
Disclosure of Invention
The embodiment of the invention aims to provide a method, a device and a storage medium for sharpening a foggy image, which can automatically extract the characteristics of the foggy image, complete the style conversion of a single foggy image and a fogless image, and do not need to acquire the foggy image and a real fogless image which are matched in pairs under the same scene for training, thereby conveniently and flexibly sharpening the foggy image.
To achieve the above object, an embodiment of the present invention provides a method for sharpening a foggy image, including the following steps:
acquiring a foggy image dataset and a fogless image dataset;
constructing a loop to generate an antagonistic network and a perceptual loss network; the loop generation countermeasure network comprises a first generator, a second generator, a first discriminator and a second discriminator;
inputting the foggy image data set and the fogless image data set into the circularly generated antagonistic network, and training by combining the perception loss network to obtain an optimal generation model;
inputting the image to be defogged into the optimal generation model to obtain a corresponding predicted fog-free image;
and carrying out Laplacian pyramid reduction on the predicted fog-free image to obtain a clear fog-free image.
Preferably, the method further comprises:
and comparing the clear fog-free image with the image to be defogged, and selecting a peak signal-to-noise ratio and structural similarity to evaluate the reliability of a test result.
Preferably, the building cycle generates an antagonistic network and a loss-aware network, and specifically includes:
constructing the first generator and the second generator, wherein the first generator and the second generator both use 9 residual blocks, each residual block is composed of two identical convolution layers, the size of each convolution layer is 3 × 3, and the number of the convolution layers is 256;
constructing the first discriminator and the second discriminator, wherein both the first discriminator and the second discriminator adopt 70 × 70 PatchGAN;
constructing the loss-aware network; wherein the loss-aware network is a VGG-16 network.
Preferably, the inputting the fog-image dataset and the fog-free image dataset into the cyclic generation antagonistic network, and training in combination with the perceptual loss network to obtain an optimal generation model specifically includes:
preprocessing a first foggy image of the foggy image data set, inputting the first foggy image into the first generator to obtain a first fogless image, and inputting the first fogless image into the second generator to obtain a second foggy image;
inputting a second fog-free image of the fog-free image dataset into the second generator to obtain a third fog-free image, and inputting the third fog-free image into the first generator to obtain a third fog-free image;
the first discriminator discriminating whether the second hazy image is from the second generator or the hazy image dataset and passing forward-facing impairments between the second hazy image and the first hazy image to the second generator to optimize a loss function of the second generator;
the second discriminator discriminating whether the third fog-free image is from the first generator or the fog-free image dataset and passing a backward contrast loss between the third fog-free image and the second fog-free image to the first generator to optimize a loss function of the first generator;
the perception loss network limits generation of a foggy image structure and a fogless image structure through forward cyclic consistent loss and forward perception cyclic consistent loss between the second foggy image and the first foggy image and backward cyclic consistent loss and backward perception cyclic consistent loss between the third fogless image and the second fogless image, so that network training efficiency is improved;
and when the discrimination probability of the second discriminator is 0.5, taking the first generator and the second generator obtained by training at the moment as optimal generation models.
Preferably, the performing laplacian pyramid reduction on the predicted fog-free image to obtain a clear fog-free image specifically includes:
setting the top layer of the Laplacian pyramid as a low-resolution fog-free image;
and performing Laplace upsampling on the low-resolution fog-free image to obtain the clear fog-free image.
Another embodiment of the present invention provides an apparatus for fog image sharpening, the apparatus including:
the data set acquisition module is used for acquiring a foggy image data set and a fogless image data set;
the network construction module is used for constructing a cyclic generation countermeasure network and a perception loss network; the loop generation countermeasure network comprises a first generator, a second generator, a first discriminator and a second discriminator;
the training module is used for inputting the foggy image data set and the fogless image data set into the circularly generated antagonistic network and training by combining the perception loss network to obtain an optimal generation model;
the testing module is used for inputting the image to be defogged into the optimal generation model to obtain a corresponding predicted fog-free image;
and the restoring module is used for carrying out Laplacian pyramid restoration on the predicted fog-free image to obtain a clear fog-free image.
The invention also provides a device using the method for clearing the foggy image, which comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor realizes the method for clearing the foggy image according to any one of the above items when executing the computer program.
Another embodiment of the present invention provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed, controls an apparatus in which the computer-readable storage medium is located to perform the method for fog image sharpening as described in any one of the above.
Compared with the prior art, the method, the device and the storage medium for clearing the foggy image provided by the embodiment of the invention have the advantages that the confrontation network and the perception loss network are generated through constructing the circulation, the characteristics of the foggy image are automatically extracted, the style conversion of the single foggy image and the fogless image is completed, the training of the pair-matched foggy image and the real fogless image in the same scene is not required, and therefore the foggy image is cleared conveniently and flexibly.
Drawings
Fig. 1 is a schematic flow chart of a method for sharpening a foggy image according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a network structure for fog image sharpening according to an embodiment of the present invention
FIG. 3 is a schematic diagram illustrating a comparison of qualitative results of defogging of a naturally occurring image according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a defogging map with a high fog concentration according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating an overall operation of a fog-based image sharpening method according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an apparatus for sharpening a foggy image according to an embodiment of the present invention;
fig. 7 is a schematic diagram of an apparatus using a method for fog image sharpening according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, a schematic flow chart of a method for sharpening a foggy image according to an embodiment of the present invention is shown, where the method includes steps S1 to S5:
s1, acquiring a foggy image data set and a fogless image data set;
s2, constructing a loop to generate an antagonistic network and a perception loss network; the loop generation countermeasure network comprises a first generator, a second generator, a first discriminator and a second discriminator;
s3, inputting the foggy image data set and the fogless image data set into the cyclic generation antagonistic network, and training by combining the perception loss network to obtain an optimal generation model;
s4, inputting the image to be defogged into the optimal generation model to obtain a corresponding predicted fog-free image;
and S5, performing Laplacian pyramid reduction on the predicted fog-free image to obtain a clear fog-free image.
Specifically, a fog-present image dataset and a fog-free image dataset are acquired. It is noted that since the training process of the present invention does not require matching of the foggy and fogless images of the same scene, it is only necessary to acquire the datasets of the foggy and fogless images without emphasizing the matching problem of the images. Preferably, the NYU-Depth dataset, the I-HAZE dataset and the O-HAZE dataset are selected, although images taken autonomously may also be selected.
Constructing a loop to generate an antagonistic network and a perceptual loss network; the loop generation countermeasure network comprises a first generator, a second generator, a first discriminator and a second discriminator;
inputting the foggy image data set and the fogless image data set into the circularly generated antagonistic network, and training by combining the perception loss network to obtain an optimal generation model;
constructing a loop to generate an antagonistic network and a perceptual loss network; the loop generation countermeasure network comprises a first generator, a second generator, a first discriminator and a second discriminator. And the loop generation countermeasure network trains a binary minimum maximum countermeasure loss and loop consistency loss combined function through an Adam algorithm to obtain an optimal generator and an optimal discriminator. For convenience of description, the first generator is denoted by Gab, the second generator is denoted by Gba, the first discriminator is denoted by Da, and the second discriminator is denoted by Db. The network structure of the present invention is composed of two parts: a cycle generation countermeasure network and a loss-aware network, and referring specifically to fig. 2, fig. 2 is a schematic diagram of a network structure for providing fog image sharpening according to this embodiment of the present invention.
And inputting the foggy image data set and the fogless image data set into a cyclic generation countermeasure network, and training by combining a perception loss network to obtain an optimal generation model. The perception loss network does not realize the matching of the input foggy image and the output foggy image as well as the input real image and the output real image on the pixel, but makes the input foggy image and the output real image similar in a characteristic space as much as possible, improves the definition of the output image and promotes the visual perception.
And inputting the image to be defogged into the optimal generation model to obtain a corresponding predicted fog-free image. The optimal generation model obtained after training is proved to be capable of generating a fog-free image from the fog image, so that the image to be defogged can be input for testing to obtain a corresponding prediction fog-free image.
The method comprises the steps of conducting Laplacian pyramid reduction on a predicted fog-free image to obtain a clear fog-free image, conducting Laplacian pyramid reduction on the obtained predicted fog-free image to enable the obtained predicted fog-free image to be clearer, determining whether Laplacian pyramid upsampling is needed or not according to the resolution of an original image, reducing the resolution of the predicted fog-free image which is output by a circularly generated countermeasure network, and omitting the process if the resolution of the original fog-free image is 256 × 256.
According to the method for clearing the foggy image, provided by the embodiment 1 of the invention, the confrontation network and the perception loss network are generated through building a cycle, the characteristics of the foggy image are automatically extracted, the style conversion of a single foggy image and a fogless image is completed, and the training of the pair-matched foggy image and the real fogless image in the same scene is not required to be obtained, so that the foggy image is conveniently and flexibly cleared.
As an improvement of the above, the method further comprises:
and comparing the clear fog-free image with the image to be defogged, and selecting a peak signal-to-noise ratio and structural similarity to evaluate the reliability of a test result.
Specifically, in order to verify the effectiveness and reliability of the method, a qualitative and quantitative evaluation mode is adopted to evaluate the effect of the test result, the reliability of the model is verified on a cross data set, namely, a clear fog-free image is compared with an image to be defogged, and the peak signal-to-noise ratio and the structural similarity are selected to evaluate the reliability of the test result. Wherein, the Peak Signal-to-Noise Ratio is also called Peak Signal-to-Noise Ratio, PSNR for short; the structural similarity is also called structural similarity, SSIM for short.
Referring to fig. 3, fig. 3 is a schematic diagram illustrating a qualitative result comparison of defogging of a natural fogging image according to the embodiment of the invention. Referring to fig. 4, fig. 4 is a schematic diagram of a defogging map with a high fog concentration according to another embodiment of the present invention.
As can be seen from fig. 3 and 4, the invention can realize the clearness of the foggy image on the natural foggy image, and the clearness effect is better when the local color tones of the foggy image are similar. Meanwhile, the model of the invention can find the shadow of the image after being cleared more easily, because the image after defogging retains the tone of the original natural foggy image.
As an improvement of the above scheme, the constructing a loop to generate an antagonistic network and a perceptual loss network specifically includes:
constructing the first generator and the second generator, wherein the first generator and the second generator both use 9 residual blocks, each residual block is composed of two identical convolution layers, the size of each convolution layer is 3 × 3, and the number of the convolution layers is 256;
constructing the first discriminator and the second discriminator, wherein both the first discriminator and the second discriminator adopt 70 × 70 PatchGAN;
constructing the loss-aware network; wherein the loss-aware network is a VGG-16 network.
Specifically, a first generator and a second generator are constructed, wherein the generator structure shows good effect on image migration and super-resolution tasks, the first generator and the second generator both use 9 residual blocks, each residual block is composed of two identical convolutional layers, the size of the convolutional kernel is 3 × 3, the number of the convolutional kernels is 256, an example regularized Re L U activation function is connected after each convolutional layer of the generator, and the flip filling mode is selected in the first layer and the last layer of the residual block and the convolutional layer because the 0 filling of the convolutional layer standard can cause serious artifact in the image conversion, see Table 1, wherein the Table 1 is the network structure of the first generator and the second generator provided by the embodiment of the invention.
TABLE 1 network architecture of first and second generators
The method comprises the steps of constructing a first discriminator and a second discriminator, wherein the first discriminator and the second discriminator both adopt 70 × 70PatchGAN, the structure has fewer parameters, and can completely process images with any size in a convolution mode, and the discriminator is widely applied to a discriminator structure for generating an anti-network, wherein the output data of the last layer of convolutional layers is one-dimensional data, the first four layers of convolutional layers are followed by an example regularized leak Re L U activation function, the negative nonzero slope is 0.2, the last layer is followed by a Sigmoid activation function, and discrimination probability can be obtained, and referring to table 2, the table 2 is the network structure of the first discriminator and the second discriminator provided by the embodiment of the invention.
TABLE 2 arbiter network architecture
The VGG-16 network model consists of five groups of convolution layers and three full-connection layers, the size of a convolution kernel is 3 × 3, the step length is 1, the network enables the space dimension of an image to be gradually reduced, the high-level image features are convenient to extract, the high-level image features have good nonlinearity, the convergence speed is high, the performance is good, the VGG-16 network is used for extracting the image features to construct a perception loss function so as to optimize the cycle generation countermeasure network, and therefore the full-connection layers are omitted.
As an improvement of the above scheme, the inputting the foggy image dataset and the fogless image dataset into the cyclic generation antagonistic network, and training in combination with the perceptual loss network to obtain an optimal generation model specifically includes:
preprocessing a first foggy image of the foggy image data set, inputting the first foggy image into the first generator to obtain a first fogless image, and inputting the first fogless image into the second generator to obtain a second foggy image;
inputting a second fog-free image of the fog-free image dataset into the second generator to obtain a third fog-free image, and inputting the third fog-free image into the first generator to obtain a third fog-free image;
the first discriminator discriminating whether the second hazy image is from the second generator or the hazy image dataset and passing forward-facing impairments between the second hazy image and the first hazy image to the second generator to optimize a loss function of the second generator;
the second discriminator discriminating whether the third fog-free image is from the first generator or the fog-free image dataset and passing a backward contrast loss between the third fog-free image and the second fog-free image to the first generator to optimize a loss function of the first generator;
the perception loss network limits generation of a foggy image structure and a fogless image structure through forward cyclic consistent loss and forward perception cyclic consistent loss between the second foggy image and the first foggy image and backward cyclic consistent loss and backward perception cyclic consistent loss between the third fogless image and the second fogless image, so that network training efficiency is improved;
and when the discrimination probability of the second discriminator is 0.5, taking the first generator and the second generator obtained by training at the moment as optimal generation models.
Specifically, a first fogging image of the fogging image data set is preprocessed and input to the first generator Gab to obtain a first fogging-free image, and the first fogging-free image is input to the second generator Gba to obtain a second fogging image. The preprocessing is to adopt a Gaussian pyramid to perform Gaussian low-pass filtering and downsampling on the first foggy image.
Inputting a second fog-free image of the fog-free image dataset into a second generator Gba to obtain a third fog-free image, and inputting the third fog-free image into a first generator Gab to obtain a third fog-free image;
the first discriminator Da discriminates whether the second foggy image is from the second generator Gba or the foggy image data set, and transfers the forward facing loss between the second foggy image and the first foggy image to the second generator Gba to optimize the loss function of the second generator Gba and improve the effect of generating the foggy image.
The second discriminator Db discriminates whether the third fog-free image is from the first generator Gab or the fog-free image data set, and transfers the backward contrast loss between the third fog-free image and the second fog-free image to the first generator Gab to optimize the loss function of the first generator Gab, improving the effect of generating the fog-free image.
The perception loss network limits generation of the foggy image structure and the fogless image structure through forward cyclic consistent loss and forward perception cyclic consistent loss between the second foggy image and the first foggy image and backward cyclic consistent loss and backward perception cyclic consistent loss between the third fogless image and the second fogless image to improve network training efficiency. The second foggy image generated by the second generator Gba and the first foggy image in the foggy image data set have forward cycle consistent loss and forward sensing cycle consistent loss, the third fogless image generated by the first generator Gab and the second fogless image in the fogless image data set have backward cycle consistent loss and backward sensing cycle consistent loss, and the foggy image and the fogless image are generated through the limitation of the two loss functions, so that the network training efficiency is improved, and the quality of the defogged image is further improved.
When the discrimination probability of the second discriminator Db is 0.5, the first generator Gab and the second generator Gba trained at this time are used as the optimal generation model. Since the first and second discriminators Da and Db only need to discriminate whether the images are the foggy image and the fogless image in the data set, that is, only need to know the pixel characteristics of the images, the training of the loop generation countermeasure network does not need the paired foggy and fogless images in the same scene. The first generator Gab continuously learns the characteristic distribution of the fog-free image of the data set, and when the second discriminator Db cannot judge whether the fog-free image is from the first generator Gab or the input fog-free image data set, namely the discrimination probability reaches 0.5, the cyclic generation confrontation network training is optimized.
In practice, the network training of the present invention is the training of the network model by a joint loss function, wherein the joint loss function is L (G)ab,Gba,Da,Db)＝LCycleGAN(Gab,Gbs,Da,Db)+γLP(Gab,Gba)，
Where φ is obtained at feature extractors at the second and fifth levels of the perceptual loss network, that is, LCycleGAN(Gab,Gbs,Da,Db) Generating a loss function for the loop that is generated during training of the countermeasure network, LP(Gab,Gba) Is a loss function generated in the training of the loss-aware network.
The joint loss function comprises cycle generation countermeasure network loss and cycle perception consistent loss, in order to achieve balance, weight gamma is weighted before cycle perception consistent loss, and the final aim is to find optimal generators Gab and Gba through training, namely:
as an improvement of the above scheme, the performing laplacian pyramid reduction on the predicted fog-free image to obtain a clear fog-free image specifically includes:
setting the top layer of the Laplacian pyramid as a low-resolution fog-free image;
and performing Laplace upsampling on the low-resolution fog-free image to obtain the clear fog-free image.
Specifically, the top layer of the laplacian pyramid is set to be a low-resolution fog-free image, the low-resolution fog-free image is subjected to laplacian upsampling, and the obtained fog-free image retains most of edge information of the fog-free image, so that the quality of the fog-free image is improved, and the clear fog-free image is obtained. That is, the predicted fog-free image is converted into a high-resolution defogged image by the laplacian pyramid.
In order to more conveniently understand the implementation process of the present invention, refer to fig. 5, which is a general workflow diagram of the fog image sharpening provided by the embodiment of the present invention, and the implementation process of the present invention can be more clearly understood from fig. 5.
Referring to fig. 6, a schematic structural diagram of an apparatus for clearing a fog image according to an embodiment of the present invention is shown, where the apparatus includes:
a dataset acquisition module 11, configured to acquire a foggy image dataset and a fogless image dataset;
the network construction module 12 is used for constructing a cyclic generation countermeasure network and a perception loss network; the loop generation countermeasure network comprises a first generator, a second generator, a first discriminator and a second discriminator;
a training module 13, configured to input the foggy image dataset and the fogless image dataset into the cyclic generation countermeasure network, and train in combination with the perceptual loss network to obtain an optimal generation model;
the test module 14 is configured to input the image to be defogged to the optimal generation model to obtain a corresponding predicted fog-free image;
and the restoring module 15 is configured to perform laplacian pyramid restoration on the predicted fog-free image to obtain a clear fog-free image.
The device for clearing the foggy image provided by the embodiment of the present invention can implement all the processes of the method for clearing the foggy image described in any one of the embodiments, and the functions and technical effects of the modules and units in the device are respectively the same as those of the method for clearing the foggy image described in the embodiment, and are not described herein again.
Referring to fig. 7, the schematic diagram of an apparatus using a method for fog image sharpening according to an embodiment of the present invention includes a processor 10, a memory 20, and a computer program stored in the memory 20 and configured to be executed by the processor 10, where the processor 10 implements the method for fog image sharpening according to any one of the above embodiments when executing the computer program.
Illustratively, the computer program may be divided into one or more modules/units, which are stored in the memory 20 and executed by the processor 10 to implement the present invention. One or more of the modules/units may be a series of computer program instruction segments capable of performing specific functions, the instruction segments being used to describe the execution of a computer program in a method for hazy image sharpness. For example, the computer program may be divided into a data set acquisition module, a network construction module, a training module, a testing module, and a recovery module, each module having the following specific functions:
a dataset acquisition module 11, configured to acquire a foggy image dataset and a fogless image dataset;
the network construction module 12 is used for constructing a cyclic generation countermeasure network and a perception loss network; the loop generation countermeasure network comprises a first generator, a second generator, a first discriminator and a second discriminator;
a training module 13, configured to input the foggy image dataset and the fogless image dataset into the cyclic generation countermeasure network, and train in combination with the perceptual loss network to obtain an optimal generation model;
the test module 14 is configured to input the image to be defogged to the optimal generation model to obtain a corresponding predicted fog-free image;
and the restoring module 15 is configured to perform laplacian pyramid restoration on the predicted fog-free image to obtain a clear fog-free image.
The device using the method for clearing the foggy image can be computing equipment such as a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The device using the method for fog image sharpening can comprise, but is not limited to, a processor and a memory. It will be understood by those skilled in the art that the schematic diagram 7 is merely an example of an apparatus using the method for fog image sharpening, and does not constitute a limitation on the apparatus using the method for fog image sharpening, and may include more or less components than those shown, or combine some components, or different components, for example, the apparatus using the method for fog image sharpening may further include an input-output device, a network access device, a bus, etc.
The processor 10 may be a Central Processing Unit (CPU), other general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. The general purpose processor may be a microprocessor or the processor 10 may be any conventional processor or the like, the processor 10 being the control center of the apparatus using the method for fog image sharpening, and various interfaces and lines connecting the various parts of the entire apparatus using the method for fog image sharpening.
The memory 20 may be used to store the computer programs and/or modules, and the processor 10 implements various functions of the apparatus using the method for fog image sharpening by running or executing the computer programs and/or modules stored in the memory 20 and calling data stored in the memory 20. The memory 20 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to program use, and the like. In addition, the memory 20 may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
Wherein the device-integrated module using the method for fog image sharpening may be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as a separate product. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium and can implement the steps of the embodiments of the method when the computer program is executed by a processor. The computer program includes computer program code, and the computer program code may be in a source code form, an object code form, an executable file or some intermediate form. The computer readable medium may include: any entity or device capable of carrying computer program code, recording medium, U.S. disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution media, and the like. It should be noted that the computer readable medium may contain other components which may be suitably increased or decreased as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, in accordance with legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunications signals.
The embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium includes a stored computer program, and when the computer program runs, the apparatus where the computer-readable storage medium is located is controlled to execute the method for sharpening the foggy image according to any of the above embodiments.
To sum up, the method, the device and the storage medium for clearing the fog images provided by the embodiment of the invention utilize the advantages of deep learning, do not need to evaluate intermediate parameters of a traditional atmospheric scattering model, automatically extract the characteristics of the fog images by constructing a loop generation countermeasure network and a perception loss network, finish the style conversion of single fog images and fog-free images, do not need to acquire the fog images and real fog-free images which are matched in pairs under the same scene, thereby conveniently and flexibly clearing the fog images, namely adding the consistent loss of loop perception on the basis of the originally loop generation countermeasure network, aiming at extracting the characteristics of low-level images and high-level images from the second pooling layer and the fifth pooling layer of a VGG-16 network, and simultaneously adding a Laplace pyramid to perform upsampling on network output images after the loop generation of the countermeasure network in order to reduce the interference of a Gaussian downsampling process on the final output fog-free images, the quality of the final output fog-free image is improved.
While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention.
Claims (8)
1. A method of fog image sharpening, comprising the steps of:
acquiring a foggy image dataset and a fogless image dataset;
constructing a loop to generate an antagonistic network and a perceptual loss network; the loop generation countermeasure network comprises a first generator, a second generator, a first discriminator and a second discriminator;
inputting the foggy image data set and the fogless image data set into the circularly generated antagonistic network, and training by combining the perception loss network to obtain an optimal generation model;
inputting the image to be defogged into the optimal generation model to obtain a corresponding predicted fog-free image;
and carrying out Laplacian pyramid reduction on the predicted fog-free image to obtain a clear fog-free image.
2. The method of fog image sharpening of claim 1, further comprising:
and comparing the clear fog-free image with the image to be defogged, and selecting a peak signal-to-noise ratio and structural similarity to evaluate the reliability of a test result.
3. The method for fog image sharpening as recited in claim 1, wherein the construction cycle generates a countering network and a perceptual loss network, specifically comprising:
constructing the first generator and the second generator, wherein the first generator and the second generator both use 9 residual blocks, each residual block is composed of two identical convolution layers, the size of each convolution layer is 3 × 3, and the number of the convolution layers is 256;
constructing the first discriminator and the second discriminator, wherein both the first discriminator and the second discriminator adopt 70 × 70 PatchGAN;
constructing the loss-aware network; wherein the loss-aware network is a VGG-16 network.
4. The method for fog-image sharpening as claimed in claim 1, wherein the inputting the fog-image dataset and the fog-free image dataset into the recurrent generation antagonistic network, and training in combination with the perceptual loss network to obtain an optimal generation model specifically comprises:
preprocessing a first foggy image of the foggy image data set, inputting the first foggy image into the first generator to obtain a first fogless image, and inputting the first fogless image into the second generator to obtain a second foggy image;
inputting a second fog-free image of the fog-free image dataset into the second generator to obtain a third fog-free image, and inputting the third fog-free image into the first generator to obtain a third fog-free image;
the first discriminator discriminating whether the second hazy image is from the second generator or the hazy image dataset and passing forward-facing impairments between the second hazy image and the first hazy image to the second generator to optimize a loss function of the second generator;
the second discriminator discriminating whether the third fog-free image is from the first generator or the fog-free image dataset and passing a backward contrast loss between the third fog-free image and the second fog-free image to the first generator to optimize a loss function of the first generator;
the perception loss network limits generation of a foggy image structure and a fogless image structure through forward cyclic consistent loss and forward perception cyclic consistent loss between the second foggy image and the first foggy image and backward cyclic consistent loss and backward perception cyclic consistent loss between the third fogless image and the second fogless image, so that network training efficiency is improved;
and when the discrimination probability of the second discriminator is 0.5, taking the first generator and the second generator obtained by training at the moment as optimal generation models.
5. The method according to claim 4, wherein the performing Laplacian pyramid reduction on the predicted fog-free image to obtain a clear fog-free image comprises:
setting the top layer of the Laplacian pyramid as a low-resolution fog-free image;
and performing Laplace upsampling on the low-resolution fog-free image to obtain the clear fog-free image.
6. An apparatus for fog image sharpening, comprising:
the data set acquisition module is used for acquiring a foggy image data set and a fogless image data set;
the network construction module is used for constructing a cyclic generation countermeasure network and a perception loss network; the loop generation countermeasure network comprises a first generator, a second generator, a first discriminator and a second discriminator;
the training module is used for inputting the foggy image data set and the fogless image data set into the circularly generated antagonistic network and training by combining the perception loss network to obtain an optimal generation model;
the testing module is used for inputting the image to be defogged into the optimal generation model to obtain a corresponding predicted fog-free image;
and the restoring module is used for carrying out Laplacian pyramid restoration on the predicted fog-free image to obtain a clear fog-free image.
7. An apparatus using a method of hazy image clarification, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, the processor when executing the computer program implementing the method of hazy image clarification according to any one of claims 1 to 5.
8. A computer-readable storage medium, comprising a stored computer program, wherein the computer program, when executed, controls an apparatus in which the computer-readable storage medium is located to perform the method for fog image sharpening as claimed in any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010193972.2A CN111507909A (en) | 2020-03-18 | 2020-03-18 | Method and device for clearing fog image and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010193972.2A CN111507909A (en) | 2020-03-18 | 2020-03-18 | Method and device for clearing fog image and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111507909A true CN111507909A (en) | 2020-08-07 |
Family
ID=71875828
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010193972.2A Pending CN111507909A (en) | 2020-03-18 | 2020-03-18 | Method and device for clearing fog image and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111507909A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112215774A (en) * | 2020-10-13 | 2021-01-12 | 北京字节跳动网络技术有限公司 | Model training and image defogging method, device, equipment and computer readable medium |
| CN112288699A (en) * | 2020-10-23 | 2021-01-29 | 北京百度网讯科技有限公司 | Method, device, equipment and medium for evaluating relative definition of image |
| CN112509091A (en) * | 2020-12-10 | 2021-03-16 | 上海联影医疗科技股份有限公司 | Medical image reconstruction method, apparatus, device and medium |
| CN112561792A (en) * | 2020-12-25 | 2021-03-26 | 北京百度网讯科技有限公司 | Image style migration method and device, electronic equipment and storage medium |
| CN112801919A (en) * | 2021-03-22 | 2021-05-14 | 恒生电子股份有限公司 | Image defogging model training method, defogging processing method and device and storage medium |
| CN112949553A (en) * | 2021-03-22 | 2021-06-11 | 陈懋宁 | Face image restoration method based on self-attention cascade generation countermeasure network |
| CN113537416A (en) * | 2021-09-17 | 2021-10-22 | 深圳市安软科技股份有限公司 | Method and related equipment for converting text into image based on generative confrontation network |
| CN113554872A (en) * | 2021-07-19 | 2021-10-26 | 昭通亮风台信息科技有限公司 | Detection early warning method and system for traffic intersection and curve |
| CN113658051A (en) * | 2021-06-25 | 2021-11-16 | 南京邮电大学 | Image defogging method and system based on cyclic generation countermeasure network |
| CN113822976A (en) * | 2021-06-08 | 2021-12-21 | 腾讯科技（深圳）有限公司 | Training method and device of generator, storage medium and electronic device |
| CN114387160A (en) * | 2022-03-23 | 2022-04-22 | 北京大甜绵白糖科技有限公司 | Training method, image processing method, device, electronic equipment and storage medium |
| CN115311138A (en) * | 2022-07-06 | 2022-11-08 | 北京科技大学 | Image super-resolution method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107578386A (en) * | 2017-09-01 | 2018-01-12 | 荆门程远电子科技有限公司 | A kind of optimization defogging processing method of unmanned plane shooting image |
| CN109300090A (en) * | 2018-08-28 | 2019-02-01 | 哈尔滨工业大学（威海） | A kind of single image to the fog method generating network based on sub-pix and condition confrontation |
| CN109472818A (en) * | 2018-10-17 | 2019-03-15 | 天津大学 | A kind of image defogging method based on deep neural network |
| CN109493303A (en) * | 2018-05-30 | 2019-03-19 | 湘潭大学 | A kind of image defogging method based on generation confrontation network |
| CN110136075A (en) * | 2019-04-18 | 2019-08-16 | 中国地质大学（武汉） | It is a kind of to recycle the remote sensing image defogging method for generating confrontation network based on edge sharpening |
-
2020
- 2020-03-18 CN CN202010193972.2A patent/CN111507909A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107578386A (en) * | 2017-09-01 | 2018-01-12 | 荆门程远电子科技有限公司 | A kind of optimization defogging processing method of unmanned plane shooting image |
| CN109493303A (en) * | 2018-05-30 | 2019-03-19 | 湘潭大学 | A kind of image defogging method based on generation confrontation network |
| CN109300090A (en) * | 2018-08-28 | 2019-02-01 | 哈尔滨工业大学（威海） | A kind of single image to the fog method generating network based on sub-pix and condition confrontation |
| CN109472818A (en) * | 2018-10-17 | 2019-03-15 | 天津大学 | A kind of image defogging method based on deep neural network |
| CN110136075A (en) * | 2019-04-18 | 2019-08-16 | 中国地质大学（武汉） | It is a kind of to recycle the remote sensing image defogging method for generating confrontation network based on edge sharpening |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112215774B (en) * | 2020-10-13 | 2023-09-29 | 抖音视界有限公司 | Model training and image defogging methods, apparatus, devices and computer readable media |
| CN112215774A (en) * | 2020-10-13 | 2021-01-12 | 北京字节跳动网络技术有限公司 | Model training and image defogging method, device, equipment and computer readable medium |
| CN112288699A (en) * | 2020-10-23 | 2021-01-29 | 北京百度网讯科技有限公司 | Method, device, equipment and medium for evaluating relative definition of image |
| CN112288699B (en) * | 2020-10-23 | 2024-02-09 | 北京百度网讯科技有限公司 | Method, device, equipment and medium for evaluating relative definition of image |
| CN112509091A (en) * | 2020-12-10 | 2021-03-16 | 上海联影医疗科技股份有限公司 | Medical image reconstruction method, apparatus, device and medium |
| CN112509091B (en) * | 2020-12-10 | 2023-11-14 | 上海联影医疗科技股份有限公司 | Medical image reconstruction method, device, equipment and medium |
| CN112561792A (en) * | 2020-12-25 | 2021-03-26 | 北京百度网讯科技有限公司 | Image style migration method and device, electronic equipment and storage medium |
| CN112561792B (en) * | 2020-12-25 | 2023-10-03 | 北京百度网讯科技有限公司 | Image style migration method and device, electronic equipment and storage medium |
| CN112801919A (en) * | 2021-03-22 | 2021-05-14 | 恒生电子股份有限公司 | Image defogging model training method, defogging processing method and device and storage medium |
| CN112949553A (en) * | 2021-03-22 | 2021-06-11 | 陈懋宁 | Face image restoration method based on self-attention cascade generation countermeasure network |
| CN113822976A (en) * | 2021-06-08 | 2021-12-21 | 腾讯科技（深圳）有限公司 | Training method and device of generator, storage medium and electronic device |
| CN113658051A (en) * | 2021-06-25 | 2021-11-16 | 南京邮电大学 | Image defogging method and system based on cyclic generation countermeasure network |
| CN113658051B (en) * | 2021-06-25 | 2023-10-13 | 南京邮电大学 | Image defogging method and system based on cyclic generation countermeasure network |
| CN113554872B (en) * | 2021-07-19 | 2022-09-20 | 昭通亮风台信息科技有限公司 | Detection early warning method and system for traffic intersection and curve |
| CN113554872A (en) * | 2021-07-19 | 2021-10-26 | 昭通亮风台信息科技有限公司 | Detection early warning method and system for traffic intersection and curve |
| CN113537416A (en) * | 2021-09-17 | 2021-10-22 | 深圳市安软科技股份有限公司 | Method and related equipment for converting text into image based on generative confrontation network |
| CN114387160A (en) * | 2022-03-23 | 2022-04-22 | 北京大甜绵白糖科技有限公司 | Training method, image processing method, device, electronic equipment and storage medium |
| CN115311138A (en) * | 2022-07-06 | 2022-11-08 | 北京科技大学 | Image super-resolution method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111507909A (en) | Method and device for clearing fog image and storage medium | |
| Qu et al. | Enhanced pix2pix dehazing network | |
| Engin et al. | Cycle-dehaze: Enhanced cyclegan for single image dehazing | |
| CN111488865B (en) | Image optimization method and device, computer storage medium and electronic equipment | |
| CN111079764B (en) | Low-illumination license plate image recognition method and device based on deep learning | |
| CN111062880A (en) | Underwater image real-time enhancement method based on condition generation countermeasure network | |
| CN114757832B (en) | Face super-resolution method and device based on cross convolution attention pair learning | |
| Zhang et al. | Single image dehazing via dual-path recurrent network | |
| CN110675334A (en) | Image enhancement method and device | |
| CN112581370A (en) | Training and reconstruction method of super-resolution reconstruction model of face image | |
| US20220414838A1 (en) | Image dehazing method and system based on cyclegan | |
| Montulet et al. | Deep learning for robust end-to-end tone mapping | |
| CN113066034A (en) | Face image restoration method and device, restoration model, medium and equipment | |
| US11887218B2 (en) | Image optimization method, apparatus, device and storage medium | |
| Ebenezer et al. | Single image haze removal using conditional wasserstein generative adversarial networks | |
| Hovhannisyan et al. | AED-Net: A single image dehazing | |
| Babu et al. | An efficient image dahazing using Googlenet based convolution neural networks | |
| Liu et al. | Facial image inpainting using multi-level generative network | |
| Tan et al. | Low-light image enhancement with geometrical sparse representation | |
| CN114119428B (en) | Image deblurring method and device | |
| Jiang et al. | Haze relevant feature attention network for single image dehazing | |
| CN116703750A (en) | Image defogging method and system based on edge attention and multi-order differential loss | |
| Holla et al. | EFID: edge-focused image denoising using a convolutional neural network | |
| Truong et al. | Depth map inpainting and super-resolution with arbitrary scale factors | |
| CN115311152A (en) | Image processing method, image processing apparatus, electronic device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |