CN116704309A - Image defogging identification method and system based on improved generation of countermeasure network - Google Patents
Image defogging identification method and system based on improved generation of countermeasure network Download PDFInfo
- Publication number
- CN116704309A CN116704309A CN202310551956.XA CN202310551956A CN116704309A CN 116704309 A CN116704309 A CN 116704309A CN 202310551956 A CN202310551956 A CN 202310551956A CN 116704309 A CN116704309 A CN 116704309A
- Authority
- CN
- China
- Prior art keywords
- image
- defogging
- network
- feature
- ambient light
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 76
- 238000013507 mapping Methods 0.000 claims description 37
- 238000012549 training Methods 0.000 claims description 29
- 238000004364 calculation method Methods 0.000 claims description 25
- 238000000605 extraction Methods 0.000 claims description 22
- 238000002834 transmittance Methods 0.000 claims description 18
- 230000004927 fusion Effects 0.000 claims description 15
- 238000009826 distribution Methods 0.000 claims description 14
- 238000005457 optimization Methods 0.000 claims description 14
- 230000004913 activation Effects 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000000149 argon plasma sintering Methods 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 3
- 239000002131 composite material Substances 0.000 claims description 2
- 230000008859 change Effects 0.000 abstract description 5
- 230000005540 biological transmission Effects 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 22
- 230000008569 process Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 230000001965 increasing effect Effects 0.000 description 6
- 238000012544 monitoring process Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 4
- 230000006872 improvement Effects 0.000 description 3
- 230000015572 biosynthetic process Effects 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000000428 dust Substances 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000000779 smoke Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0475—Generative networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/094—Adversarial learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/30—Noise filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an image defogging identification method and system based on an improved generation countermeasure network, wherein the method is used for extracting features of images to output feature images with different scales, generating a map which is fixedly input to ambient light and used for estimating pseudo ambient light, simultaneously using the same input to generate the map of scattered ambient light, splicing the generated map and an input defogging image to form a new pseudo defogging image, judging whether the pseudo defogging image is a real image, and reversely calculating the transmissivity of an atmospheric scattering model and the ambient light value to obtain the defogging image by using the calculated parameters in an actual scene. The invention can make the network extract more image characteristics so as to better estimate the sound transmission rate and the ambient light value, and simultaneously the color loss module can avoid the phenomenon of color change of the image generation result, solve the problems of image information distortion and color change of image identification in the current image defogging link, and improve the accuracy of image defogging identification.
Description
Technical Field
The invention relates to the technical field of countermeasure network models, in particular to an image defogging identification method and system for generating a countermeasure network based on improvement.
Background
At present, along with the rapid change of environmental factors, the frequency of weather such as fog and haze in a plurality of areas is increased, and the weather greatly influences the fields of satellite remote sensing monitoring, target identification and tracking, traffic monitoring and the like, so that the image defogging has very important practical application value in the aspects; in addition, due to the existence of haze and fog, the color saturation and contrast in the image are reduced, and a lot of image details are lost. Image defogging is also of extremely high research interest.
The traditional image defogging method is used for estimating the transmissivity and the ambient light value by using a convolution network structure so as to restore the defogging image, the traditional method is used for omitting the context information of some images in the process of estimating the transmissivity and the ambient light value, and the method is only based on the principle of generating the ignored foggy image from the aspect of characteristic processing of the images. Therefore, the conventional method is prone to causing the problem of defogging image identification that the defogging effect of the image is not obvious and the accuracy of the defogging image identification is not high.
Disclosure of Invention
The present invention aims to solve at least one of the technical problems in the related art to some extent.
To this end, the invention proposes an image defogging recognition method based on an improved generation of an countermeasure network. The mapping relation between the fog-free image and the fog-free image can be directly learned, and the added depth residual error connection in the network model can enable the network to extract more image features so as to better estimate the sound transmission rate and the ambient light value, meanwhile, the color loss can avoid the phenomenon that the color of the image generation result changes, the problems of image information distortion and color change in the image defogging link are solved, and the image defogging recognition precision is improved.
Another object of the invention is to propose a defogging method for images based on an improved generation of an countermeasure network.
To achieve the above object, an aspect of the present invention provides an image defogging recognition method based on an improved generation of an countermeasure network, including:
acquiring a training data set containing fog image samples and corresponding non-fog image samples, and constructing a defogging identification network model so as to perform model training on the defogging identification network model by utilizing the training data set; the defogging identification network model comprises a generation network model and a discrimination network model, wherein the generation network model comprises a feature extraction network and a feature enhancement network, and the feature enhancement network comprises a residual error connection network and a multi-head attention network;
inputting the training data set into the generating network model, and performing a first feature classification operation on the features of the hazy image sample and the corresponding non-hazy image sample by using the feature extraction network to obtain a multi-scale feature map; inputting the multi-scale feature map into the feature enhancement network, obtaining a fused feature map by utilizing a second feature classification operation of fusing the multi-scale feature map by the residual error connection network and the multi-head attention network, generating a mapping of an atmospheric ambient light image and a scattered atmospheric ambient light image based on the fused feature map, and calculating the transmissivity and the ambient light value of an atmospheric scattering model according to the characteristic parameters of the mapping to generate a clear haze-free image sample;
inputting the clear defogging image sample into the discrimination network model, performing third feature classification operation on the mapping and the clear defogging image sample by using a preset splicing method to obtain a pseudo defogging image, performing loss calculation on the pseudo defogging image and the defogging image sample by using a loss function, and optimizing model parameters of the defogging identification network model by using an optimization objective function and a loss calculation result to obtain a trained defogging identification network model;
and inputting the new foggy image to be identified into the trained defogging identification network model for image defogging identification so as to obtain a defogging identification result of the clear foggy image corresponding to the new foggy image.
In addition, the image defogging recognition method based on the improved generation countermeasure network according to the above embodiment of the present invention may further have the following additional technical features:
further, in one embodiment of the present invention, the feature enhancement network employs a cascaded network; inputting the multi-scale feature map into the feature enhancement network to obtain a fused feature map by using the residual error connection network and a second feature classification operation of fusing the multi-scale feature map by using the multi-head attention network, wherein the method comprises the following steps:
inputting the multi-scale feature map to the cascade network to extract initial feature information of the multi-scale feature map using a channel-by-channel convolution of 5*5;
extracting multi-scale context information of the initial characteristic information through the channel-by-channel cavity convolution of 7*7;
and performing convolution operation output on the multi-scale context information by utilizing point-by-point convolution of 1*1 to obtain the fusion characteristic map.
Further, in one embodiment of the present invention, the stitching method includes a method of adding pixels by pixel synthesis; the discriminant network model comprises a plurality of convolution layers, wherein each convolution layer uses ReLu as an activation function; adding a color consistency loss to the loss function, the loss function:
wherein p represents a pixel, ANGLE is an ANGLE calculation function, y is a foggy image, G B (G A (x) G) and G A (G B (y)) is a generated pseudo-hazy image.
Further, in an embodiment of the present invention, the calculating the transmittance of the atmospheric scattering model and the ambient light value according to the mapped feature parameter to generate a clear haze-free image sample includes:
calculation formula of atmospheric light scattering:
I(x)＝J(x)t(x)+A(x)(1-t(x))
wherein A is 0 Is the light of the atmospheric environment, A 1 M is scattered atmospheric environment light 0 、M 1 The method comprises the steps that a mapping relation matrix generated by a generation module is formed by a background image to an atmospheric ambient light image and a scattered atmospheric ambient light image, x refers to coordinates of each pixel point of the image, J (x) is a clear fog-free image, I (x) is an actually obtained fog image, A is an ambient light value of an image shooting place, and t (x) represents ambient light transmittance;
J(x)＝M 0 [I(x)+b(x)]
wherein:
further, in an embodiment of the present invention, the generated map includes a first map for estimating a pseudo-ambient light value and a second map for estimating a transmittance, the first map being a map of the fusion feature map to the atmospheric ambient light image, including: constructing a fusion feature map for generating a map of atmospheric ambient light for the input image N (x) to estimate the atmospheric environmentOptical image A 0 (x)：
The second mapping is a mapping from the fusion feature map to a scattered atmospheric ambient light image, including: fusion of the feature map based identical input image N (x) to scattered ambient light image A 1 (x) Is mapped to:
from the estimation A 0 And t (x) and I (x), and solving J (x) through a formula to obtain a defogging image.
Further, in one embodiment of the present invention, the optimization objective of generating the network model is:
wherein Div represents the difference between the two distributions, z represents the input noise data, and obeys the distribution P z True data x obeys distribution P data The judging network model D is used for classifying the input data, namely judging whether the input data belongs to real data or generated data, and the optimizing target of the judging network model is as follows:
wherein V (G, D) is defined as:
wherein E represents a data distribution desire, the defogging identifies an optimization objective function of the network model based on generating the network model and discriminating an optimization objective of the network model:
to achieve the above object, another aspect of the present invention provides an image defogging recognition system based on an improved generation of an countermeasure network, comprising:
the identification model construction module is used for acquiring a training data set containing fog image samples and corresponding non-fog image samples, and constructing a defogging identification network model so as to carry out model training on the defogging identification network model by utilizing the training data set; the defogging identification network model comprises a generation network model and a discrimination network model, wherein the generation network model comprises a feature extraction network and a feature enhancement network, and the feature enhancement network comprises a residual error connection network and a multi-head attention network;
the first feature classification module is used for inputting the training data set into the generating network model so as to perform a first feature classification operation on the features of the foggy image sample and the corresponding non-foggy image sample by utilizing the feature extraction network to obtain a multi-scale feature map; inputting the multi-scale feature map into the feature enhancement network, obtaining a fused feature map by utilizing a second feature classification operation of fusing the multi-scale feature map by the residual error connection network and the multi-head attention network, generating a mapping of an atmospheric ambient light image and a scattered atmospheric ambient light image based on the fused feature map, and calculating the transmissivity and the ambient light value of an atmospheric scattering model according to the characteristic parameters of the mapping to generate a clear haze-free image sample;
the second feature classification module is used for inputting the clear defogging image sample into the discrimination network model, performing third feature classification operation on the mapping and the clear defogging image sample by using a preset splicing method to obtain a pseudo defogging image, performing loss calculation on the pseudo defogging image and the defogging image sample by using a loss function, and optimizing model parameters of the defogging identification network model by using an optimization objective function and a loss calculation result to obtain a trained defogging identification network model;
and the image defogging recognition module is used for inputting a new defogging image to be recognized into the trained defogging recognition network model for image defogging recognition so as to obtain a defogging recognition result of a clear defogging image corresponding to the new defogging image.
The image defogging identification method and system based on the improved generation countermeasure network can quickly obtain defogging images, reduce network calculation amount, and improve defogging speed and image identification efficiency of the network on the premise of guaranteeing defogging effect.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The foregoing and/or additional aspects and advantages of the invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a flow chart of a method of image defogging identification based on an improved generation of an countermeasure network in accordance with an embodiment of the present invention;
FIG. 2 is a block diagram of an image defogging method based on an improvement in generating an countermeasure network in accordance with an embodiment of the present invention;
FIG. 3 is a schematic diagram of the architecture of a network model according to an embodiment of the invention;
FIG. 4 is a schematic diagram of a feature enhanced network architecture according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a discrimination network according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an image defogging recognition system based on an improved generation countermeasure network according to an embodiment of the present invention.
Detailed Description
It should be noted that, without conflict, the embodiments of the present invention and features of the embodiments may be combined with each other. The invention will be described in detail below with reference to the drawings in connection with embodiments.
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
Image defogging recognition methods and systems based on an improved generation of an countermeasure network according to embodiments of the present invention are described below with reference to the accompanying drawings.
Fig. 1 is a flow chart of an image defogging recognition method based on an improved generation of an countermeasure network according to an embodiment of the present invention.
As shown in fig. 1, the image defogging recognition method based on the improved generation countermeasure network includes:
s1, acquiring a training data set containing fog image samples and corresponding non-fog image samples, and constructing a defogging identification network model so as to perform model training on the defogging identification network model by using the training data set; the defogging identification network model comprises a generation network model and a discrimination network model, wherein the generation network model comprises a feature extraction network and a feature enhancement network, and the feature enhancement network comprises a residual error connection network and a multi-head attention network;
s2, inputting the training data set into a generating network model, and performing first feature classification operation on the features of the foggy image sample and the corresponding foggy image sample by utilizing a feature extraction network to obtain a multi-scale feature map; inputting the multi-scale feature map into a feature enhancement network, obtaining a fused feature map by utilizing a second feature classification operation of fusing the multi-scale feature map by a residual error connection network and a multi-head attention network, generating a mapping of an atmospheric ambient light image and a scattered atmospheric ambient light image based on the fused feature map, and calculating the transmissivity and the ambient light value of an atmospheric scattering model according to the mapped feature parameters to generate a clear haze-free image sample;
s3, inputting the clear defogging image sample into a discrimination network model, performing third feature classification operation on the mapping and the clear defogging image sample by using a preset splicing method to obtain a pseudo defogging image, performing loss calculation on the pseudo defogging image and the defogging image sample by using a loss function, and optimizing model parameters of a defogging identification network model by using an optimization objective function and a loss calculation result to obtain a trained defogging identification network model;
s4, inputting the new foggy image to be identified into the trained defogging identification network model for image defogging identification so as to obtain a defogging identification result of the clear foggy image corresponding to the new foggy image.
According to the image defogging identification method based on the improved generation countermeasure network, the image defogging identification precision can be improved by enhancing the feature extraction capability of the image, increasing the overall image details, and performing corresponding mapping of two different images by using a mapping method, and directly generating a clear image according to the mapping parameters.
As shown in fig. 2 and 3, the defogging speed of the image can be effectively improved on the premise of ensuring defogging performance for the structure diagram and the network structure diagram of the method. Firstly, the invention is divided into a training stage and a testing stage, a training set is constructed in the training stage and is input into a defogging network, the defogging generation network comprises a feature extraction structure and a feature enhancement structure, the feature mapping relation between a foggy image and a foggy image is learned, a residual error connection structure and a multi-head attention structure are added in the feature enhancement, the whole image is subjected to feature splicing, mapping, convolution and pooling operation, the refractive index of atmospheric light and the ambient light value are estimated, finally, the foggy image is restored into a clear image by an image mapping restoration method, and the clear image is input into a judging module for judgment. The network structure in the test stage tends to be balanced, and the defogged image is directly input into the network, so that the defogged image can be obtained. The defogging method provided by the invention can avoid fog residue and detail information loss in the image defogging process, and can be applied in various environments.
In one embodiment of the present invention, a feature enhancement network, as shown in fig. 4, inputs the multi-scale feature map to a cascading network to extract initial feature information of the multi-scale feature map using a channel-by-channel convolution of 5*5; extracting multi-scale context information of the initial characteristic information through the channel-by-channel cavity convolution of 7*7; and performing convolution operation on the multi-scale context information by utilizing point-by-point convolution of 1*1 to output the fusion characteristic map.
Specifically, the convolution kernel sizes of three convolution layers in the feature extraction network are 7x7x64, 3x3x128 and 3x3x512, the step sizes are 1, 2 and 2, respectively, and zero padding is selected. The convolution kernel of 7x7 can enlarge the receptive field of the network to the image, the network can better capture the detailed information of the image, and then the convolution kernels with the size of 3x3 can effectively reduce the size of the extracted feature images, thereby being beneficial to network training. The concatenated residual network blocks may be used to increase the depth of the generator network and thereby increase the capability of feature extraction.
It can be understood that the cascade residual block transfers information to a deeper layer of the network in a jump link manner, so that the characteristic response of the background and the image details is enhanced, and the image characteristic information is prevented from being lost. The network operation parameter quantity and the calculation amount can be reduced under the condition that the receptive fields are the same, the network layer number is deepened, and the network efficiency is improved to a certain extent.
In one embodiment of the invention, the feature enhancement network uses 1X1 convolution to reduce the number of channels that splice the feature map.
In one embodiment of the invention, the generation of the mapping relationship is partly a picture-to-ambient light picture feature extraction network mapping for estimating pseudo-ambient light illumination, and is also to construct a mapping given the same input image to a scattered ambient light image for estimating transmittance. And carrying out normalization operation on the feature map, and solving the ambient light parameters and the refractive index of each image.
Further, the invention generates a mapping relation M from a background image to an atmospheric ambient light image after scattering 0 、M 1 Through A 0 And A is a 1 The relation of (2) is solved to obtain a transmittance graph, namely:
it is known that whether the image is a foggy image to foggy image or a foggy image to foggy image, it can be written as a structure solved based on a transformation matrix and a bias matrix, such as:
I(x)＝J(x)t(x)+A(x)(1-t(x))
J(x)＝M 0 [I(x)+b(x)]
wherein:
to restore a clear original image, we have to obtain the ambient light and transmittance of the image at the location where it is located, and thus construct a mapping network for generating ambient light from a known input image to estimate the generated ambient light image a 0 (x) Can be written in the following form:
to estimate the transmittance, we also construct a given identical input image N (x) to the scattered ambient light image A according to the above formula 1 (x) Has the following form:
considering the strong feature extraction capability of the convolutional neural network, the two mapping networks are realized by a backbone network CNN network, and after atmospheric environment light and transmittance are estimated, the conversion from a foggy image to a foggy image is finally realized according to a formula.
I(x)＝J(x)t(x)+A(1-t(x))
Where x refers to the coordinates of each pixel point of the image, J (x) is a clear haze-free image, I (x) is an actually obtained haze image, a is an ambient light value of the image capturing area, and t (x) is expressed as transmittance.
Further, the generated network model and the discrimination network model realize learning through countermeasure, and finally reach Nash equilibrium points, so that both parties can reach the optimal. The generating network model is mainly used for generating data which accords with expectations and minimizing the distribution difference of the generated data and the real data, and the optimizing target of the generating network model is as follows:
wherein Div represents the difference between the two distributions, z represents the input noise data, subject to the distribution P z True data x obeys distribution P data . The judging network model block D is used for classifying the input data, namely judging whether the input data belongs to real data or generated data, and obtaining the optimizing target of the judging network model as follows:
wherein V (G, D) is defined as:
where E represents the data distribution desire. Comprehensively generating a network model and judging an optimization target of the network model, and finally obtaining an optimization target function of the whole defogging recognition network model:
further, as shown in fig. 5, the discrimination network model of the present invention mainly adopts a binary network to discriminate the synthesized pseudo fog image and the true fog image, and the closer to the true fog image, the environment light parameter and the transmittance estimated by the network are to the true value. The output value of the judging network guides the training of the generating network, so that the generating network can generate more realistic environmental parameters in the learning iteration process, the convergence of the network is improved, and meanwhile, the judging network also continuously updates the network parameters in each iteration, and the obvious characteristics between the generated image and the real hazy image are distinguished, so that the self identification capability is improved. And iterating in a circulating way until the discrimination network cannot distinguish the generation network generation, and balancing the discrimination network generation. The structure of the discriminant network model is shown in fig. 5, where the network consists of 3 convolution modules, all using ReLu as the activation function. The convolution kernel size of the first 2 convolution layers is 3x3 and is used for extracting the output image characteristics of the generated model, finally, the convolution layers with the convolution kernel size of 1x1 and the step length of 1 are added to serve as one-dimensional output, a Sigmoid activation function is used for judging whether the generated image accords with real sample distribution, and color consistency loss is added in a loss function.
Where p represents a pixel and ANGLE is an ANGLE calculation function that calculates the ANGLE difference between two colors, and RGB can be regarded as a three-dimensional vector for the colors. y is a hazy image. G B (G A (x) G) and G A (G B (y)) is an image obtained by splicing the atmospheric light value generated by the generation module and the haze-free image. Calculation G A (G B And (y)) and accumulating the included angle between the color vector of each pixel in the image y, and can effectively judge whether the parameters generated by the generating module approach to the true value.
Further, the transmittance of the trained network model, estimated from the atmospheric ambient light image and the scattered ambient light image generated by the generator, is independent of the foggy image sample.
Further, the ambient light and the transmittance of the atmosphere are almost unchanged in a certain time, so that the transmittance and the ambient light value are estimated by the anti-deduction calculation module according to the mapping relation, and the clear fog-free image can be reversely solved under the condition that the fog image is known.
In summary, the present invention treats the foggy image as a non-linear composite of the foggy image and the foggy image. A light generator network is used for generating a map which is fixedly input to ambient light and is used for estimating pseudo ambient light, a generator is used for generating a map which is scattered by the same input, finally, an atmospheric scattering model is utilized for carrying out inverse solution to obtain a defogging image, a scene requiring real-time defogging is met, compared with the traditional method for directly generating the defogging image by using a generating network, the defogging image can be obtained by estimating the transmissivity through a simple mathematical relation, the calculated amount of the network is reduced, and the defogging speed of the network is improved on the premise of ensuring the defogging effect.
The image defogging identification method based on the improved generation countermeasure network can be applied to a plurality of computer vision systems such as cell monitoring, intelligent traffic, civil aviation assistance, post-disaster rescue, remote sensing observation, automatic driving and the like, for example, for the cell monitoring, the image defogging identification method based on the improved generation countermeasure network model and the image defogging method can effectively process images of people entering and exiting the cell in foggy weather and haze weather in autumn and winter, so that the problem of personnel omission caused by bad weather is avoided, and the capability of the monitoring system for coping with the bad weather is effectively improved. For intelligent traffic, the improved generation countermeasure network model and the image defogging method can accurately identify and detect the running condition and the illegal behavior of the vehicle in foggy days, and can improve the effectiveness of a traffic supervision system in foggy days monitoring. For civil aviation assistance, the image information or video information in the aircraft running process is input into the improved generation countermeasure network model and the image defogging method, so that related images can be accurately processed in cloud and fog, the visibility of the civil aviation aircraft in the high altitude can be improved in the aircraft landing process, and the observation capability of the civil aviation aircraft in the cloud and fog is further improved. In the post-disaster rescue aspect, the image information in the fire disaster can be rebuilt by adopting the improved generation countermeasure network model and the image defogging method, and the foggy fire disaster image is converted into clear image information, so that the rescue efficiency of rescue in the fire disaster can be improved. For remote sensing observation, the improved generation countermeasure network model and the image defogging method can be adopted to process the fog map generated by the refraction of the natural light of the atmosphere, so that the observation efficiency and the environment adaptability of the remote sensing observation system are improved. For automatic driving, the improved generation countermeasure network model and the image defogging method can accurately process road condition information of the haze weather in real time, solve the problem that the judgment of a driver is affected due to low weather visibility, and further guarantee safe driving of the driver in the haze weather. Therefore, the improved generation countermeasure network model and the image defogging method have wide application range, the foggy image or the image containing smoke dust is input into the model, the problems of detail loss and color change in the image defogging process can be relieved through the improvement of the model, and meanwhile, the global information defogging effect of the image is improved on the basis of not losing the detection speed.
According to the image defogging method based on the improved generation countermeasure network, feature fusion and splicing are carried out on the feature graphs with different layers, so that the semantic information capacity of the up-sampling feature graph is increased, on one hand, the depth of the network is increased, and on the other hand, the extraction capacity of the network on the global features is enhanced. Meanwhile, the adopted cavity convolution method is expected to have fewer parameters than the common convolution method, so that the calculated amount of parameters of the network is reduced, and the running speed of the network is increased. According to the invention, the feature fusion enhancement branch is added, and the cavity convolution method and the feature residual error module are added, so that the receptive field is enlarged, the global detail parameter feature extraction capability of the model is enhanced, and the defogging recognition efficiency of the image is improved, so that the defogging precision of the image generation is effectively improved.
In order to implement the above-described embodiment, as shown in fig. 6, there is also provided in the present embodiment an image defogging recognition system 10 based on an improved generation of an countermeasure network, the system 10 comprising:
the recognition model construction module 100 is configured to acquire a training data set including a fog image sample and a corresponding non-fog image sample, and construct a defogging recognition network model, so as to perform model training on the defogging recognition network model by using the training data set; the defogging identification network model comprises a generation network model and a discrimination network model, wherein the generation network model comprises a feature extraction network and a feature enhancement network, and the feature enhancement network comprises a residual error connection network and a multi-head attention network;
the first feature classification module 200 is configured to input the training data set into the generated network model, so as to perform a first feature classification operation on features of the hazy image sample and the corresponding non-hazy image sample by using the feature extraction network to obtain a multi-scale feature map; inputting the multi-scale feature map into the feature enhancement network, obtaining a fused feature map by utilizing a second feature classification operation of fusing the multi-scale feature map by using a residual error connection network and a multi-head attention network, generating a mapping of an atmospheric ambient light image and a scattered atmospheric ambient light image based on the fused feature map, and calculating the transmissivity and the ambient light value of an atmospheric scattering model according to the mapped feature parameters to generate a clear haze-free image sample;
the second feature classification module 300 is configured to input a clear and haze-free image sample into the discrimination network model, perform a third feature classification operation on the mapped and clear and haze-free image sample by using a preset stitching method to obtain a pseudo haze image, perform loss calculation on the pseudo haze image and the haze image sample by using a loss function, and optimize model parameters of the haze-removal recognition network model by using an optimization objective function and a loss calculation result to obtain a trained haze-removal recognition network model;
the image defogging recognition module 400 is configured to input a new defogging image to be recognized into the trained defogging recognition network model for image defogging recognition, so as to obtain a defogging recognition result of a clear defogging image corresponding to the new defogging image.
Further, the feature enhancement network adopts a cascade network; the first feature classification module 200 is further configured to:
inputting the multi-scale feature map to the cascade network to extract initial feature information of the multi-scale feature map by utilizing a channel-by-channel convolution of 5*5;
extracting multi-scale context information of the initial characteristic information through the channel-by-channel cavity convolution of 7*7;
and performing convolution operation output on the multi-scale context information by utilizing the point-by-point convolution of 1*1 to obtain a fusion feature map.
Further, the stitching method comprises a method of adopting pixel-to-pixel synthesis addition; discriminating a network model comprising a plurality of convolution layers, wherein each convolution layer uses ReLu as an activation function; adding a color consistency loss in a loss function, the loss function:
wherein p represents a pixel, ANGLE is an ANGLE calculation function, y is a foggy image, G B (G A (x) G) and G A (G B (y)) is a generated pseudo-hazy image.
Further, the second feature classification module 300 is further configured to:
calculation formula of atmospheric light scattering:
I(x)＝J(x)t(x)+A(x)(1-t(x))
wherein A is 0 Is the light of the atmospheric environment, A 1 M is scattered atmospheric environment light 0 、M 1 The method comprises the steps that a mapping relation matrix generated by a generation module is formed by a background image to an atmospheric ambient light image and a scattered atmospheric ambient light image, x refers to coordinates of each pixel point of the image, J (x) is a clear fog-free image, I (x) is an actually obtained fog image, A is an ambient light value of an image shooting place, and t (x) represents ambient light transmittance;
J(x)＝M 0 [I(x)+b(x)]
wherein:
according to the image defogging recognition system based on the improved generation countermeasure network, feature fusion and splicing are carried out on the feature images with different layers, so that the semantic information capacity of the up-sampling feature images is improved. The method increases the depth of the network on one hand and enhances the extraction capability of the network on the global characteristics on the other hand. Meanwhile, the adopted cavity convolution method is expected to have fewer parameters than the common convolution method, so that the calculated amount of parameters of the network is reduced, and the running speed of the network is increased. According to the invention, the feature fusion enhancement branch is added by designing, and the cavity convolution method and the feature residual error module are added, so that the receptive field is enlarged, the feature extraction capability of the model on the global detail parameters is enhanced, and the defogging recognition precision of the image is effectively improved.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "plurality" means at least two, for example, two, three, etc., unless specifically defined otherwise.
Claims (10)
1. An image defogging recognition method based on improved generation of an countermeasure network, comprising:
acquiring a training data set containing fog image samples and corresponding non-fog image samples, and constructing a defogging identification network model so as to perform model training on the defogging identification network model by utilizing the training data set; the defogging identification network model comprises a generation network model and a discrimination network model, wherein the generation network model comprises a feature extraction network and a feature enhancement network, and the feature enhancement network comprises a residual error connection network and a multi-head attention network;
inputting the training data set into the generating network model, and performing a first feature classification operation on the features of the hazy image sample and the corresponding non-hazy image sample by using the feature extraction network to obtain a multi-scale feature map; inputting the multi-scale feature map into the feature enhancement network, obtaining a fused feature map by utilizing a second feature classification operation of fusing the multi-scale feature map by the residual error connection network and the multi-head attention network, generating a mapping of an atmospheric ambient light image and a scattered atmospheric ambient light image based on the fused feature map, and calculating the transmissivity and the ambient light value of an atmospheric scattering model according to the characteristic parameters of the mapping to generate a clear haze-free image sample;
inputting the clear defogging image sample into the discrimination network model, performing third feature classification operation on the mapping and the clear defogging image sample by using a preset splicing method to obtain a pseudo defogging image, performing loss calculation on the pseudo defogging image and the defogging image sample by using a loss function, and optimizing model parameters of the defogging identification network model by using an optimization objective function and a loss calculation result to obtain a trained defogging identification network model;
and inputting the new foggy image to be identified into the trained defogging identification network model for image defogging identification so as to obtain a defogging identification result of the clear foggy image corresponding to the new foggy image.
2. The method of claim 1, wherein the feature enhancement network employs a cascaded network; inputting the multi-scale feature map into the feature enhancement network to obtain a fused feature map by using the residual error connection network and a second feature classification operation of fusing the multi-scale feature map by using the multi-head attention network, wherein the method comprises the following steps:
inputting the multi-scale feature map to the cascade network to extract initial feature information of the multi-scale feature map using a channel-by-channel convolution of 5*5;
extracting multi-scale context information of the initial characteristic information through the channel-by-channel cavity convolution of 7*7;
and performing convolution operation output on the multi-scale context information by utilizing point-by-point convolution of 1*1 to obtain the fusion characteristic map.
3. The method of claim 1, wherein the stitching method comprises a method of pixel-to-pixel synthetic addition; the discriminant network model comprises a plurality of convolution layers, wherein each convolution layer uses ReLu as an activation function; adding a color consistency loss to the loss function, the loss function:
wherein p represents a pixel, ANGLE is an ANGLE calculation function, y is a foggy image, G B (G A (x) G) and G A (G B (y)) is a generated pseudo-hazy image.
4. A method according to claim 3, wherein said calculating the transmittance of the atmospheric scattering model and the ambient light value from the mapped feature parameters to generate a clear haze-free image sample comprises:
calculation formula of atmospheric light scattering:
I(x)＝J(x)t(x)+A(x)(1-t(x))
wherein A is 0 Is the light of the atmospheric environment, A 1 M is scattered atmospheric environment light 0 、M 1 The method comprises the steps that a mapping relation matrix generated by a generation module is formed by a background image to an atmospheric ambient light image and a scattered atmospheric ambient light image, x refers to coordinates of each pixel point of the image, J (x) is a clear fog-free image, I (x) is an actually obtained fog image, A is an ambient light value of an image shooting place, and t (x) represents ambient light transmittance;
J(x)＝M 0 [I(x)+b(x)]
wherein:b (x) ＝-A(x)[1-t(x)]。
5. the method of claim 4, wherein the generated map comprises a first map for estimating pseudo-ambient light values and a second map for estimating transmittance, the first map being a map of the fused feature map to the atmospheric ambient light image, comprising: constructing a fused feature map to generate a map of the ambient light for the input image N (x) to estimate the ambient light image A 0 (x)：
The second mapping is a mapping from the fusion feature map to a scattered atmospheric ambient light image, including: fusion of the feature map based identical input image N (x) to scattered ambient light image A 1 (x) Is mapped to:
from the estimation A 0 And t (x) and I (x), and solving J (x) through a formula to obtain a defogging image.
6. The method of claim 5, wherein the optimization objective of generating the network model is:
wherein Div represents the difference between the two distributions, z represents the input noise data, and obeys the distribution P z True data x obeys distribution P data The judging network model D is used for classifying the input data, namely judging whether the input data belongs to real data or generated data, and the optimizing target of the judging network model is as follows:
wherein V (G, D) is defined as:
wherein E represents a data distribution desire, the defogging identifies an optimization objective function of the network model based on generating the network model and discriminating an optimization objective of the network model:
7. an image defogging recognition system based on an improved generation of an countermeasure network, comprising:
the identification model construction module is used for acquiring a training data set containing fog image samples and corresponding non-fog image samples, and constructing a defogging identification network model so as to carry out model training on the defogging identification network model by utilizing the training data set; the defogging identification network model comprises a generation network model and a discrimination network model, wherein the generation network model comprises a feature extraction network and a feature enhancement network, and the feature enhancement network comprises a residual error connection network and a multi-head attention network;
the first feature classification module is used for inputting the training data set into the generating network model so as to perform a first feature classification operation on the features of the foggy image sample and the corresponding non-foggy image sample by utilizing the feature extraction network to obtain a multi-scale feature map; inputting the multi-scale feature map into the feature enhancement network, obtaining a fused feature map by utilizing a second feature classification operation of fusing the multi-scale feature map by the residual error connection network and the multi-head attention network, generating a mapping of an atmospheric ambient light image and a scattered atmospheric ambient light image based on the fused feature map, and calculating the transmissivity and the ambient light value of an atmospheric scattering model according to the characteristic parameters of the mapping to generate a clear haze-free image sample;
the second feature classification module is used for inputting the clear defogging image sample into the discrimination network model, performing third feature classification operation on the mapping and the clear defogging image sample by using a preset splicing method to obtain a pseudo defogging image, performing loss calculation on the pseudo defogging image and the defogging image sample by using a loss function, and optimizing model parameters of the defogging identification network model by using an optimization objective function and a loss calculation result to obtain a trained defogging identification network model;
and the image defogging recognition module is used for inputting a new defogging image to be recognized into the trained defogging recognition network model for image defogging recognition so as to obtain a defogging recognition result of a clear defogging image corresponding to the new defogging image.
8. The system of claim 7, wherein the feature enhancement network employs a cascaded network; the first feature classification module is further configured to:
inputting the multi-scale feature map to the cascade network to extract initial feature information of the multi-scale feature map using a channel-by-channel convolution of 5*5;
extracting multi-scale context information of the initial characteristic information through the channel-by-channel cavity convolution of 7*7;
and performing convolution operation output on the multi-scale context information by utilizing point-by-point convolution of 1*1 to obtain the fusion characteristic map.
9. The system of claim 7, wherein the stitching method comprises a method of pixel-to-pixel composite addition; the discriminant network model comprises a plurality of convolution layers, wherein each convolution layer uses ReLu as an activation function; adding a color consistency loss to the loss function, the loss function:
wherein p represents a pixel, ANGLE is an ANGLE calculation function, y is a foggy image, G B (G A (x) G) and G A (G B (y)) is a generated pseudo-hazy image.
10. The system of claim 9, wherein the second feature classification module is further configured to:
calculation formula of atmospheric light scattering:
I(x)＝J(x)t(x)+A(x)(1-t(x))
wherein A is 0 Is the light of the atmospheric environment, A 1 M is scattered atmospheric environment light 0 、M 1 The method comprises the steps that a mapping relation matrix generated by a generation module is formed by a background image to an atmospheric ambient light image and a scattered atmospheric ambient light image, x refers to coordinates of each pixel point of the image, J (x) is a clear fog-free image, I (x) is an actually obtained fog image, A is an ambient light value of an image shooting place, and t (x) represents ambient light transmittance;
J(x)＝M 0 [I(x)+b(x)]
wherein:b (x) ＝-A(x)[1-t(x)]。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310551956.XA CN116704309A (en) | 2023-05-16 | 2023-05-16 | Image defogging identification method and system based on improved generation of countermeasure network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310551956.XA CN116704309A (en) | 2023-05-16 | 2023-05-16 | Image defogging identification method and system based on improved generation of countermeasure network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116704309A true CN116704309A (en) | 2023-09-05 |
Family
ID=87838328
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310551956.XA Pending CN116704309A (en) | 2023-05-16 | 2023-05-16 | Image defogging identification method and system based on improved generation of countermeasure network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116704309A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117911507A (en) * | 2023-12-28 | 2024-04-19 | 武汉志远智能控制技术有限公司 | Molten iron liquid level identification method, device, equipment and storage medium |
-
2023
- 2023-05-16 CN CN202310551956.XA patent/CN116704309A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117911507A (en) * | 2023-12-28 | 2024-04-19 | 武汉志远智能控制技术有限公司 | Molten iron liquid level identification method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109584248B (en) | Infrared target instance segmentation method based on feature fusion and dense connection network | |
| CN106910175B (en) | Single image defogging algorithm based on deep learning | |
| CN109886312B (en) | Bridge vehicle wheel detection method based on multilayer feature fusion neural network model | |
| CN109086668B (en) | Unmanned aerial vehicle remote sensing image road information extraction method based on multi-scale generation countermeasure network | |
| CN112270249A (en) | Target pose estimation method fusing RGB-D visual features | |
| CN111489301B (en) | Image defogging method based on image depth information guide for migration learning | |
| CN111461083A (en) | Rapid vehicle detection method based on deep learning | |
| CN110246181B (en) | Anchor point-based attitude estimation model training method, attitude estimation method and system | |
| CN111611861B (en) | Image change detection method based on multi-scale feature association | |
| CN110717863B (en) | Single image snow removing method based on generation countermeasure network | |
| CN113887349A (en) | Road area image identification method based on image and point cloud fusion network | |
| Chen et al. | ASF-Net: Adaptive screening feature network for building footprint extraction from remote-sensing images | |
| Qiang et al. | Forest fire smoke detection under complex backgrounds using TRPCA and TSVB | |
| CN113052106A (en) | Airplane take-off and landing runway identification method based on PSPNet network | |
| Ogunrinde et al. | A review of the impacts of defogging on deep learning-based object detectors in self-driving cars | |
| CN114998566A (en) | Interpretable multi-scale infrared small and weak target detection network design method | |
| CN113989261A (en) | Unmanned aerial vehicle visual angle infrared image photovoltaic panel boundary segmentation method based on Unet improvement | |
| CN116704309A (en) | Image defogging identification method and system based on improved generation of countermeasure network | |
| Sun et al. | IRDCLNet: Instance segmentation of ship images based on interference reduction and dynamic contour learning in foggy scenes | |
| Qian et al. | CIASM-Net: a novel convolutional neural network for dehazing image | |
| CN112767267A (en) | Image defogging method based on simulation polarization fog-carrying scene data set | |
| CN116740516A (en) | Target detection method and system based on multi-scale fusion feature extraction | |
| CN116524189A (en) | High-resolution remote sensing image semantic segmentation method based on coding and decoding indexing edge characterization | |
| CN114913337A (en) | Camouflage target frame detection method based on ternary cascade perception | |
| Zhao et al. | Image dehazing based on haze degree classification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |