CN113658072A - Underwater image enhancement method based on progressive feedback network - Google Patents

Underwater image enhancement method based on progressive feedback network Download PDF

Info

Publication number
CN113658072A
CN113658072A CN202110935907.7A CN202110935907A CN113658072A CN 113658072 A CN113658072 A CN 113658072A CN 202110935907 A CN202110935907 A CN 202110935907A CN 113658072 A CN113658072 A CN 113658072A
Authority
CN
China
Prior art keywords
network
image
stage
underwater
attention
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110935907.7A
Other languages
Chinese (zh)
Other versions
CN113658072B (en
Inventor
牛玉贞
张宇杰
张凌昕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fuzhou University
Original Assignee
Fuzhou University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fuzhou University filed Critical Fuzhou University
Priority to CN202110935907.7A priority Critical patent/CN113658072B/en
Publication of CN113658072A publication Critical patent/CN113658072A/en
Application granted granted Critical
Publication of CN113658072B publication Critical patent/CN113658072B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20048—Transform domain processing
    • G06T2207/20064—Wavelet transform [DWT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/30—Assessment of water resources

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Biophysics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Processing (AREA)

Abstract

The invention provides an underwater image enhancement method based on a progressive feedback network, which comprises the following steps: step S1: pairing underwater image data for training, and then performing data enhancement and normalization processing on the underwater image data to obtain paired images to be trained; step S2: inputting paired images to be trained into a multi-stage progressive image enhancement network capable of enhancing images at each stage by combining discrete wavelet transformation and an attention feedback mechanism, training an image enhancement model capable of enhancing underwater images, and correcting the images at each stage of the network by using a supervision attention module; step S3: setting a target loss function of the image enhancement network; step S4: using the paired training image to enhance network convergence to Nash balance; step S5: normalization processing is carried out on the underwater image to be enhanced, then the trained image enhancement model is input, and the enhanced image is output; the invention is beneficial to improving the underwater image quality.

Description

Underwater image enhancement method based on progressive feedback network
Technical Field
The invention relates to the technical field of image processing and computer vision, in particular to an underwater image enhancement method based on a progressive feedback network.
Background
Underwater image enhancement techniques are gaining attention due to their importance in the fields of ocean engineering and underwater robotics. The quality of underwater imaging has a great influence on underwater operation, and tasks such as submarine exploration, underwater target detection and the like which depend on vision have high requirements on the quality of underwater images, and the efficiency and the accuracy of the tasks are seriously reduced due to the low-quality underwater images. Enhancement of underwater images is a challenging problem due to the complexity of the underwater environment and lighting conditions. Generally, underwater images are subject to wavelength dependent absorption and scattering, including forward scattering, which is a scattering phenomenon in which light reflected from objects in the water is offset by a small angle when transmitted to a camera, and backscattering, resulting in blurred image details. Backscattering refers to impurities in the water that are scattered and received directly by the camera when illuminating objects in the water, resulting in low image contrast. Furthermore, in the ocean, many plankton, plants, silt and dust, etc. float up and down, introducing noise and increasing the effect of scattering. These adverse effects reduce visibility, reduce contrast, and even introduce color bias, which causes severe degradation of the quality of the underwater image and prevents marine engineering and underwater operations.
The existing underwater image enhancement methods are mainly divided into two types: firstly, a method based on deep learning regards the conversion from an underwater image to a normal image as a mapping relation, and learns the mapping change by using the learning fitting capability of a network, so as to realize the conversion from the underwater image to the normal image, but the method has higher requirements on the size and the category of data volume, and is usually accompanied by the coding and decoding operation on the image in the learning process, and some image information loss may be generated in the process, so that the local detail blurring phenomenon occurs in the enhanced image. And secondly, a method using a physical model is adopted, a mathematical model of the degradation process of the underwater image is established firstly, and a clear underwater image is obtained through inversion of the degradation process, and the method needs to estimate model parameters. However, the underwater environment is complex, conditions such as illumination are variable, the estimation of parameters is difficult and the precision is low, so that the quality of the enhanced image is generally low, and meanwhile, different models to be established are different due to different factors considered by different environments, so that the methods have great limitations.
The existing method usually has information loss in the image enhancement process, so that the enhanced image is easy to have a detail blurring phenomenon. The method comprises the steps of dividing an image enhancement process into a plurality of stages, carrying out independent enhancement on an image in each stage, carrying out feedback correction on the output of each stage and a label image, inputting the corrected result into the next stage, realizing gradual optimization and promotion, and remarkably improving the quality of an underwater image.
Disclosure of Invention
The invention provides an underwater image enhancement method based on a progressive feedback network, which is beneficial to improving the quality of underwater images.
The invention adopts the following technical scheme.
An underwater image enhancement method based on a progressive feedback network comprises the following steps:
step S1: pairing underwater image data for training, and then performing data enhancement and normalization processing on the underwater image data to obtain paired images to be trained;
step S2: inputting paired images to be trained into a multi-stage progressive image enhancement network capable of enhancing images at each stage by combining discrete wavelet transformation and an attention feedback mechanism, training an image enhancement model capable of enhancing underwater images, and correcting the images at each stage of the network by using a supervision attention module;
step S3: setting a target loss function of the image enhancement network;
step S4: using the paired training image to enhance network convergence to Nash balance;
step S5: and carrying out normalization processing on the underwater image to be enhanced, then inputting the trained image enhancement model, and outputting the enhanced image.
The step S1 includes the steps of:
step S11: matching the underwater image for training with the corresponding label image;
step S12: carrying out uniform random turning operation on all paired images to be trained, and enhancing data;
step S13: all images to be trained are normalized, an image I (I, j) is given, and the normalized image is
Figure BDA0003213130440000021
At pixel position (i, j), a normalized value is calculated
Figure BDA0003213130440000022
The formula of (1) is as follows:
Figure BDA0003213130440000023
wherein, (i, j) represents the position of the pixel, and the normalized paired image is used as the input image and label image pair of the subsequent step.
The step S2 includes step S21, step S22, step S23;
the step S21 specifically includes: designing a multi-stage progressive image enhancement network, wherein the input of the network is a normalized underwater image
Figure BDA0003213130440000031
Outputting the underwater image after being enhanced; the network is divided into three stages to be progressively executed, each stage is combined with a discrete wavelet transform and an attention feedback mechanism to enhance the image, the network structures of the three stages are the same, and a supervision attention module is used between the stages to supervise the characteristics of the stage, namely the supervision attention module is used to supervise the image enhanced at the stage after the first stage and the second stage.
Three of the progressive executionIn each stage, the first stage inputs underwater image characteristics F1 after a convolutional layer with a convolutional kernel of 3x3 and a step length of 1inThe output of the first stage is the image feature F1 enhanced by the current stageoutThe feature is compared with the normalized underwater image
Figure BDA0003213130440000032
The output of the supervision attention module is the corrected feature F2 as an input to the supervision attention modulein；
The second stage input is the corrected feature F2inThe output of the second stage is the image feature F2 enhanced by the current stageoutThe output features are corrected by the supervision attention module to obtain image features F3 in the same way as in the previous stageinTaking the corrected features as input of the third stage;
the output of the third stage is the image feature F3 enhanced by the current stageoutObtaining a final enhanced image R after a convolution layer with a convolution kernel of 3x3 and a step length of 1, namely the output of the image enhancement network; the calculation formula is as follows:
Figure 100002_1
Figure BDA0003213130440000034
Figure BDA0003213130440000035
Figure BDA0003213130440000036
Fiout＝Neti(Fiin) I is 1, 2, 3 formula six;
wherein
Figure BDA0003213130440000037
Representing the normalized underwater input image, SAM (X) representing the network of supervising attention modules, Neti(1), i ═ 1, 2, 3 denote three stage networks in a multi-stage progressive image enhancement network;
the step S22 specifically includes: designing the three-stage network in the step S21, wherein the three-stage network has the same structure; each stage network can be divided into three layers from top to bottom, and each layer consists of a wavelet pooling layer, a residual attention module, a wavelet anti-pooling layer and an attention feedback module.
The wavelet pooling layer decomposes features using discrete Haar wavelets, using four decomposition kernels LLT，LHT，HLT，HHTWherein the low-frequency and high-frequency filters are respectively
Figure BDA0003213130440000041
Figure BDA0003213130440000042
The wavelet anti-pooling layer uses discrete Haar wavelets to carry out combined reconstruction on the low-frequency component and the high-frequency component, and the used anti-pooling nuclear parameters are the same as those of the wavelet anti-pooling layer;
the residual attention module consists of convolution with two layers of convolution kernels of 3x3 and step length of 1 and a channel attention network, and the calculation formula is as follows:
Xout＝ECA_Net(ADD[Xin，Relu(Conv(Relu(Conv(Xin))))]) A formula of nine;
wherein XinIndicating input features, XoutRepresenting output characteristics, Relu being an activation function, ADD representing characteristic addition operation, ECA _ Net (x) representing a channel attention network;
the attention feedback module consists of a channel attention network and a sigmoid function, and the calculation formula is as follows
W＝Sigmoid(ECA_Net(Xin) Equation ten;
wherein XinRepresenting input features, W representing output feature weights, ECA _ Net (, x) representing the channel attention network.
The network structure of each stage of the multi-stage progressive image enhancement network is the same, namely the input of the first level of each stage of the network is the input of the current stage of the network, namely the image characteristic F1in，
When the first stage of the multi-stage progressive image enhancement network works, the method comprises the following steps:
step A1, performing forward wavelet decomposition from the first layer to the third layer to obtain low-frequency component and high-frequency component, ll1，lh1，hl1，hh1,, wherein ll1For low frequency components, lh1，hl1，hh1Are all high frequency components; will ll1As input of the second level, the low-frequency component ll is decomposed into a low-frequency component ll through a wavelet pooling layer as same as the first level2And a high frequency component lh2，hl2，hh2(ii) a Will ll2As the input of the third level, the low-frequency component ll is decomposed into the low-frequency components through the wavelet pooling layer, which is the same as the first two levels3And a high frequency component lh3，hl3，hh3；
Step A2, enhancing the features reversely through a residual attention module, a wavelet inverse pooling layer and an attention feedback module; firstly, the low-frequency component ll of the third level3And a high frequency component lh3，hl3，hh3Obtaining transformed low-frequency components ll through residual attention modules respectively4And a high frequency component lh4，hl4，hh4Inputting the transformed low-frequency component and high-frequency component into a wavelet anti-pooling layer for combined reconstruction, wherein the output characteristics of the wavelet anti-pooling layer are used as the input of an attention feedback module, and the output of the attention feedback module is a third level to a second level ll2Feedback weight w3 of vector to convert the low-frequency component ll of the second level2Multiplying the feedback weight w3 to perform correction to obtain a corrected low-frequency component ll of the second layer2′；
Step A3, correcting ll2' component and second-level high-frequency component lh2，hl2，hh2Obtaining transformed low-frequency component ll by inputting residual attention module5And a high frequency component lh5，hl5，hh5(ii) a Inputting the transformed low-frequency component and high-frequency component into a wavelet anti-pooling layer for combined reconstruction, wherein the output characteristics of the wavelet anti-pooling layer are used as the input of an attention feedback module, and the output of the attention feedback module is the second level to the first level ll1Feedback weight w2 of vector, and low-frequency component ll of first level1Multiplying the feedback weight w2 to perform correction to obtain a corrected low-frequency component ll of the second layer1′；
Step A4, correcting ll1' component and first-level high-frequency component lh1，hl1，hh1Obtaining transformed low-frequency component ll by inputting residual attention module6And a high frequency component lh6，hl6，hh6(ii) a Inputting the transformed low-frequency component and high-frequency component into a wavelet anti-pooling layer for combined reconstruction, taking the output characteristics of the wavelet anti-pooling layer as the input of an attention feedback module, taking the output of the attention feedback module as the feedback weight w1 of the first level to the initial input characteristics of the network, and inputting the initial input characteristics of the network F1inMultiplying by weight w1 to obtain the image feature F1 after the current stage is enhancedoutNamely the output of the first-stage network;
in the above way, the output F2 of the second stage network is further obtainedoutAnd the output of the second stage network F3out。
Step S23 specifically includes: designing the network of the supervised attention Module in step S21, the network input being the enhanced image features output by the first or second stage network F1outOr F2outAnd normalized underwater images
Figure BDA0003213130440000051
Taking the first module for supervision attention as an example, the input image features F1 are first inputoutObtaining a residual image through convolution with convolution kernel of 3x3 and step length of 1, and adding the residual image and the underwater image according to pixels to obtain a stage underwater enhanced image R1The obtained stage underwater enhanced image R1Calculating a loss with the normalized tag image G according to step S3; then, the stage underwater enhanced image R1R is obtained as an input to the convolutional layer with a convolution kernel of 3x3 and a step size of 11And obtaining a feature weight w4 through a sigmoid function, wherein the feature weight is used as a guide weight for the network input image feature of the supervision attention module. I.e. the originally input enhanced image feature F1outMultiplying by weight w4 to obtain corrected enhanced image feature F2inAs output from the network of supervising attention modules. The calculation formula is as follows:
Figure BDA0003213130440000061
Figure BDA0003213130440000062
enhanced image features F1 for first and second stage network outputout and F2outRespectively obtaining corrected characteristics F2 after passing through the network of the supervision attention modulein and F3inAnd underwater enhanced image R in the generation stage1 and R2。
The step S3 includes the following steps;
step S31: designing a network target loss function, wherein the network total target loss function is as follows:
l＝λ1·l1+λ2·lsa formula thirteen;
wherein ,l1 and lsAre each L1Loss and grid loss, λ1 and λ2Is each loss balance coefficient, is a real number dot product operation; the specific calculation formula of each loss is as follows:
l1＝||R-G||1+||R1-G||1+||R2-G||1a formula fourteen;
wherein R is the final output result of the designed underwater image enhancement network, R1 and R2Stage underwater enhanced images generated in the two monitoring attention modules are respectively, G is a normalized label image, | |1Is an absolute value operation;
Figure BDA0003213130440000063
wherein
Figure BDA0003213130440000071
Representing the Frobenius square norm, j representing the j-th layer, phi representing the feature extraction network, here using a pre-trained VGG16 network,
Figure BDA0003213130440000072
a Gram matrix representing the j-th layer activation characteristics extracted by the network phi is defined as follows:
Figure BDA0003213130440000073
where x represents the input image and phi represents the feature extraction network, here a pre-trained VGG16 network is usedj(x)h，w，c′Represents the j-th layer activation feature, phi, of the input image x obtained in the feature extraction network phij(x)h，w，c′Representing the transpose of the j-th layer of activation features obtained by the input image x in the feature extraction network phi, C, C' representing the number of feature channels, h representing the feature height, w representing the feature width, Cj×Hj×WjThe size of the features is activated for layer j.
The step S4 includes the following steps;
step S41: randomly dividing the matched underwater images and the label images into a plurality of batches, wherein each batch comprises N pairs of images;
step S42: inputting the underwater image into the image enhancement network in the step S2 to obtain a final enhanced image and a stage underwater enhanced image;
step S43, calculating the gradient of each parameter in the image enhancement network by using a back propagation method according to the total target loss function of the image enhancement network, and updating the parameter of the image enhancement network by using a random gradient descent method;
step S44: and (4) repeating the steps from S41 to S43 image enhancement network training by taking batches as units until the target loss function value of the image enhancement network converges to Nash balance, and storing the network parameters to finish the training process of the image enhancement network.
Compared with the prior art, the image enhanced by the existing water system image enhancement method often has a detail blurring phenomenon, the invention provides an underwater image enhancement network based on progressive feedback, which can effectively reduce information loss in an image transmission process, retain image detail information, avoid detail blurring, and be suitable for most complex scenes.
The invention has the beneficial effects that: the method is suitable for enhancing the underwater images in various complex environments, and can effectively restore the distorted colors of the images, remove the image blur and improve the contrast and brightness of the images in use. The enhanced image conforms to human subjective visual perception.
Drawings
The invention is described in further detail below with reference to the following figures and detailed description:
FIG. 1 is a schematic flow chart of an implementation of the method of the present invention;
FIG. 2 is a schematic diagram of a network model architecture in an embodiment of the invention;
FIG. 3 is a schematic diagram of a phase network model structure according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating a supervised attention network model architecture in an embodiment of the present invention;
FIG. 5 is a block diagram of a residual attention module in a phase network according to an embodiment of the present invention.
Detailed Description
As shown in the figure, the underwater image enhancement method based on the progressive feedback network comprises the following steps:
step S1: pairing underwater image data for training, and then performing data enhancement and normalization processing on the underwater image data to obtain paired images to be trained;
step S2: inputting paired images to be trained into a multi-stage progressive image enhancement network capable of enhancing images at each stage by combining discrete wavelet transformation and an attention feedback mechanism, training an image enhancement model capable of enhancing underwater images, and correcting the images at each stage of the network by using a supervision attention module;
step S3: setting a target loss function of the image enhancement network;
step S4: using the paired training image to enhance network convergence to Nash balance;
step S5: and carrying out normalization processing on the underwater image to be enhanced, then inputting the trained image enhancement model, and outputting the enhanced image.
The step S1 includes the steps of:
step S11: matching the underwater image for training with the corresponding label image;
step S12: carrying out uniform random turning operation on all paired images to be trained, and enhancing data;
step S13: all images to be trained are normalized, an image I (I, j) is given, and the normalized image is
Figure BDA0003213130440000081
At pixel position (i, j), a normalized value is calculated
Figure BDA0003213130440000082
The formula of (1) is as follows:
Figure BDA0003213130440000091
wherein, (i, j) represents the position of the pixel, and the normalized paired image is used as the input image and label image pair of the subsequent step.
The step S2 includes step S21, step S22, step S23;
the step S21 specifically includes: designing a multi-stage progressive image enhancement network, wherein the input of the network is a normalized underwater image
Figure BDA0003213130440000092
Outputting the underwater image after being enhanced; the network is divided into three stages to be progressively executed, each stage is combined with a discrete wavelet transform and an attention feedback mechanism to enhance the image, the network structures of the three stages are the same, and a supervision attention module is used between the stages to supervise the characteristics of the stage, namely the supervision attention module is used to supervise the image enhanced at the stage after the first stage and the second stage.
In the three stages of the progressive execution, the first stage inputs underwater image characteristics F1 after passing through a convolutional layer with a convolutional kernel of 3x3 and a step length of 1inThe output of the first stage is the image feature F1 enhanced by the current stageoutThe feature is compared with the normalized underwater image
Figure BDA0003213130440000097
The output of the supervision attention module is the corrected feature F2 as an input to the supervision attention modulein；
The second stage input is the corrected feature F2inThe output of the second stage is the image feature F2 enhanced by the current stageoutThe output features are corrected by the supervision attention module to obtain image features F3 in the same way as in the previous stageinTaking the corrected features as input of the third stage;
the output of the third stage is the image feature F3 enhanced by the current stageoutObtaining a final enhanced image R after a convolution layer with a convolution kernel of 3x3 and a step length of 1, namely the output of the image enhancement network; the calculation formula is as follows:
Figure 2
Figure BDA0003213130440000094
Figure BDA0003213130440000095
Figure BDA0003213130440000096
Fiout＝Neti(Fiin) I is 1, 2, 3 formula six;
wherein
Figure BDA0003213130440000101
Representing the normalized underwater input image, SAM (X) representing the network of supervising attention modules, Neti(1), i ═ 1, 2, 3 denote three stage networks in a multi-stage progressive image enhancement network;
the step S22 specifically includes: designing the three-stage network in the step S21, wherein the three-stage network has the same structure; each stage network can be divided into three layers from top to bottom, and each layer consists of a wavelet pooling layer, a residual attention module, a wavelet anti-pooling layer and an attention feedback module.
The wavelet pooling layer decomposes features using discrete Haar wavelets, using four decomposition kernels LLT，LHT，HLT，HHTWherein the low-frequency and high-frequency filters are respectively
Figure BDA0003213130440000102
Figure BDA0003213130440000103
The wavelet anti-pooling layer uses discrete Haar wavelets to carry out combined reconstruction on the low-frequency component and the high-frequency component, and the used anti-pooling nuclear parameters are the same as those of the wavelet anti-pooling layer;
the residual attention module consists of convolution with two layers of convolution kernels of 3x3 and step length of 1 and a channel attention network, and the calculation formula is as follows:
Xout＝ECA_Net(ADD[Xin，Relu(Conv(Relu(Conv(Xin))))]) A formula of nine;
wherein XinIndicating input features, XoutRepresenting output characteristics, Relu is an activation function, AD represents characteristic addition operation, and ECA _ Net (x) represents a channel attention network;
the attention feedback module consists of a channel attention network and a sigmoid function, and the calculation formula is as follows
W＝Sigmoid(ECA_Net(Xin) Equation ten;
wherein XinRepresenting input features, W representing output feature weights, ECA _ Net (, x) representing the channel attention network.
The network structure of each stage of the multi-stage progressive image enhancement network is the same, namely the input of the first level of each stage of the network is the input of the current stage of the network, namely the image characteristic F1in，
When the first stage of the multi-stage progressive image enhancement network works, the method comprises the following steps:
step A1, performing forward wavelet decomposition from the first layer to the third layer to obtain low-frequency component and high-frequency component, ll1，lh1，hl1，hh1,, wherein ll1For low frequency components, lh1，hl1，hh1Are all high frequency components; will ll1As input of the second level, the low-frequency component ll is decomposed into a low-frequency component ll through a wavelet pooling layer as same as the first level2And a high frequency component lh2，hl2，hh2(ii) a Will ll2As input to the third level, the wavelet is passed, as in the first two levelsDecomposition of pooling layer into low frequency components ll3And a high frequency component lh3，hl3，hh3；
Step A2, enhancing the features reversely through a residual attention module, a wavelet inverse pooling layer and an attention feedback module; firstly, the low-frequency component ll of the third level3And a high frequency component lh3，hl3，hh3Obtaining transformed low-frequency components ll through residual attention modules respectively4And a high frequency component lh4，hl4，hh4Inputting the transformed low-frequency component and high-frequency component into a wavelet anti-pooling layer for combined reconstruction, wherein the output characteristics of the wavelet anti-pooling layer are used as the input of an attention feedback module, and the output of the attention feedback module is a third level to a second level ll2Feedback weight w3 of vector to convert the low-frequency component ll of the second level2Multiplying the feedback weight w3 to perform correction to obtain a corrected low-frequency component ll of the second layer2′；
Step A3, correcting ll2' component and second-level high-frequency component lh2，hl2，hh2Obtaining transformed low-frequency component ll by inputting residual attention module5And a high frequency component lh5，hl5，hh5(ii) a Inputting the transformed low-frequency component and high-frequency component into a wavelet anti-pooling layer for combined reconstruction, wherein the output characteristics of the wavelet anti-pooling layer are used as the input of an attention feedback module, and the output of the attention feedback module is the second level to the first level ll1Feedback weight w2 of vector, and low-frequency component ll of first level1Multiplying the feedback weight w2 to perform correction to obtain a corrected low-frequency component ll of the second layer1′；
Step A4, correcting ll1' component and first-level high-frequency component lh1，hl1，hh1Obtaining transformed low-frequency component ll by inputting residual attention module6And a high frequency component lh6，hl6，hh6(ii) a Inputting the transformed low-frequency component and high-frequency component into wavelet anti-pooling layer for combined reconstruction, and outputting the wavelet anti-pooling layerThe characteristics are used as the input of an attention feedback module, the output of the attention feedback module is feedback weight w1 of the first level to the initial input characteristics of the network, and the initial input characteristics of the network are F1inMultiplying by weight w1 to obtain the image feature F1 after the current stage is enhancedoutNamely the output of the first-stage network;
in the above way, the output F2 of the second stage network is further obtainedoutAnd the output of the second stage network F3out。
Step S23 specifically includes: designing the network of the supervised attention Module in step S21, the network input being the enhanced image features output by the first or second stage network F1outOr F2outAnd normalized underwater images
Figure BDA0003213130440000121
Taking the first module for supervision attention as an example, the input image features F1 are first inputoutObtaining a residual image through convolution with convolution kernel of 3x3 and step length of 1, and adding the residual image and the underwater image according to pixels to obtain a stage underwater enhanced image R1The obtained stage underwater enhanced image R1Calculating a loss with the normalized tag image G according to step S3; then, the stage underwater enhanced image R1R is obtained as an input to the convolutional layer with a convolution kernel of 3x3 and a step size of 11And obtaining a feature weight w4 through a sigmoid function, wherein the feature weight is used as a guide weight for the network input image feature of the supervision attention module. I.e. the originally input enhanced image feature F1outMultiplying by weight w4 to obtain corrected enhanced image feature F2inAs output from the network of supervising attention modules. The calculation formula is as follows:
Figure BDA0003213130440000122
Figure BDA0003213130440000123
enhanced image features F1 for first and second stage network outputout and F2outRespectively obtaining corrected characteristics F2 after passing through the network of the supervision attention modulein and F3inAnd underwater enhanced image R in the generation stage1 and R2。
The step S3 includes the following steps;
step S31: designing a network target loss function, wherein the network total target loss function is as follows:
l＝λ1·l1+λ2·lsa formula thirteen;
wherein ,l1 and lsAre each L1Loss and grid loss, λ1 and λ2Is each loss balance coefficient, is a real number dot product operation; the specific calculation formula of each loss is as follows:
l1＝||R-G||1+||R1-G||1+||R2-G||1a formula fourteen;
wherein R is the final output result of the designed underwater image enhancement network, R1 and R2Stage underwater enhanced images generated in the two monitoring attention modules are respectively, G is a normalized label image, | |1Is an absolute value operation;
Figure BDA0003213130440000131
wherein
Figure BDA0003213130440000134
Representing the Frobenius square norm, j representing the j-th layer, phi representing the feature extraction network, here using a pre-trained VGG16 network,
Figure BDA0003213130440000132
a Gram matrix representing the j-th layer activation characteristics extracted by the network phi is defined as follows:
Figure BDA0003213130440000133
where x represents the input image and phi represents the feature extraction network, here a pre-trained VGG16 network is usedj(x)h，w，cRepresents the j-th layer activation feature, phi, of the input image x obtained in the feature extraction network phij(x)h，w，c′Representing the transpose of the j-th layer of activation features obtained by the input image x in the feature extraction network phi, C, C' representing the number of feature channels, h representing the feature height, w representing the feature width, Cj×Hj×WjThe size of the features is activated for layer j.
The step S4 includes the following steps;
step S41: randomly dividing the matched underwater images and the label images into a plurality of batches, wherein each batch comprises N pairs of images;
step S42: inputting the underwater image into the image enhancement network in the step S2 to obtain a final enhanced image and a stage underwater enhanced image;
step S43, calculating the gradient of each parameter in the image enhancement network by using a back propagation method according to the total target loss function of the image enhancement network, and updating the parameter of the image enhancement network by using a random gradient descent method;
step S44: and (4) repeating the steps from S41 to S43 image enhancement network training by taking batches as units until the target loss function value of the image enhancement network converges to Nash balance, and storing the network parameters to finish the training process of the image enhancement network.
The above are preferred embodiments of the present invention, and all changes made according to the technical scheme of the present invention that produce functional effects do not exceed the scope of the technical scheme of the present invention belong to the protection scope of the present invention.

Claims (10)

1. An underwater image enhancement method based on a progressive feedback network is characterized in that: the method comprises the following steps:
step S1: pairing underwater image data for training, and then performing data enhancement and normalization processing on the underwater image data to obtain paired images to be trained;
step S2: inputting paired images to be trained into a multi-stage progressive image enhancement network capable of enhancing images at each stage by combining discrete wavelet transformation and an attention feedback mechanism, training an image enhancement model capable of enhancing underwater images, and correcting the images at each stage of the network by using a supervision attention module;
step S3: setting a target loss function of the image enhancement network;
step S4: using the paired training image to enhance network convergence to Nash balance;
step S5: and carrying out normalization processing on the underwater image to be enhanced, then inputting the trained image enhancement model, and outputting the enhanced image.
2. The underwater image enhancement method based on the progressive feedback network as claimed in claim 1, wherein: the step S1 includes the steps of:
step S11: matching the underwater image for training with the corresponding label image;
step S12: carrying out uniform random turning operation on all paired images to be trained, and enhancing data;
step S13: all images to be trained are normalized, an image I (I, j) is given, and the normalized image is
Figure FDA0003213130430000011
At pixel position (i, j), a normalized value is calculated
Figure FDA0003213130430000012
The formula of (1) is as follows:
Figure FDA0003213130430000013
wherein, (i, j) represents the position of the pixel, and the normalized paired image is used as the input image and label image pair of the subsequent step.
3. The underwater image enhancement method based on the progressive feedback network as claimed in claim 1, wherein: the step S2 includes step S21, step S22, step S23;
the step S21 specifically includes: designing a multi-stage progressive image enhancement network, wherein the input of the network is a normalized underwater image
Figure FDA0003213130430000014
Outputting the underwater image after being enhanced; the network is divided into three stages to be progressively executed, each stage is combined with a discrete wavelet transform and an attention feedback mechanism to enhance the image, the network structures of the three stages are the same, and a supervision attention module is used between the stages to supervise the characteristics of the stage, namely the supervision attention module is used to supervise the image enhanced at the stage after the first stage and the second stage.
4. The underwater image enhancement method based on the progressive feedback network as claimed in claim 3, wherein: in the three stages of the progressive execution, the first stage inputs underwater image characteristics F1 after passing through a convolutional layer with a convolutional kernel of 3x3 and a step length of 1inThe output of the first stage is the image feature F1 enhanced by the current stageoutThe feature is compared with the normalized underwater image
Figure FDA0003213130430000021
The output of the supervision attention module is the corrected feature F2 as an input to the supervision attention modulein；
The second stage input is the corrected feature F2inThe output of the second stage is the image feature F2 enhanced by the current stageoutThe output features are corrected by the supervision attention module to obtain image features as in the previous stageF3inTaking the corrected features as input of the third stage;
the output of the third stage is the image feature F3 enhanced by the current stageoutObtaining a final enhanced image R after a convolution layer with a convolution kernel of 3x3 and a step length of 1, namely the output of the image enhancement network; the calculation formula is as follows:
Figure 1
Figure FDA0003213130430000023
Figure FDA0003213130430000024
Figure FDA0003213130430000025
Fiout＝Neti(Fiin) I is 1, 2, 3 formula six;
wherein
Figure FDA0003213130430000026
Representing the normalized underwater input image, SAM (X) representing the network of supervising attention modules, Neti(x), i ═ 1, 2, 3 denotes three stage networks in a multi-stage progressive image enhancement network.
5. The underwater image enhancement method based on the progressive feedback network as claimed in claim 3, wherein: the step S22 specifically includes: designing the three-stage network in the step S21, wherein the three-stage network has the same structure; each stage network can be divided into three layers from top to bottom, and each layer consists of a wavelet pooling layer, a residual attention module, a wavelet anti-pooling layer and an attention feedback module.
6. The underwater image enhancement method based on the progressive feedback network as claimed in claim 5, wherein: the wavelet pooling layer decomposes features using discrete Haar wavelets, using four decomposition kernels LLT，LHT，HLT，HHTWherein the low-frequency and high-frequency filters are respectively
Figure FDA0003213130430000031
Figure FDA0003213130430000032
The wavelet anti-pooling layer uses discrete Haar wavelets to carry out combined reconstruction on the low-frequency component and the high-frequency component, and the used anti-pooling nuclear parameters are the same as those of the wavelet anti-pooling layer;
the residual attention module consists of convolution with two layers of convolution kernels of 3x3 and step length of 1 and a channel attention network, and the calculation formula is as follows:
Xout＝ECA_Net(ADD[Xin，Relu(Conv(Relu(Conv(Xin))))]) A formula of nine;
wherein XinIndicating input features, XoutRepresenting output characteristics, Relu being an activation function, ADD representing characteristic addition operation, ECA _ Net (x) representing a channel attention network;
the attention feedback module consists of a channel attention network and a sigmoid function, and the calculation formula is as follows
W＝Sigmoid(ECA_Net(Xin) Equation ten;
wherein XinRepresenting input features, W representing output feature weights, ECA _ Net (, x) representing the channel attention network.
7. Underwater image enhancement method based on progressive feedback network according to claim 6The method is characterized in that: the network structure of each stage of the multi-stage progressive image enhancement network is the same, namely the input of the first level of each stage of the network is the input of the current stage of the network, namely the image characteristic F1in，
When the first stage of the multi-stage progressive image enhancement network works, the method comprises the following steps:
step A1, performing forward wavelet decomposition from the first layer to the third layer to obtain low-frequency component and high-frequency component, ll1，lh1，hl1，hh1,, wherein ll1For low frequency components, lh1，hl1，hh1Are all high frequency components; will ll1As input of the second level, the low-frequency component ll is decomposed into a low-frequency component ll through a wavelet pooling layer as same as the first level2And a high frequency component lh2，hl2，hh2(ii) a Will ll2As the input of the third level, the low-frequency component ll is decomposed into the low-frequency components through the wavelet pooling layer, which is the same as the first two levels3And a high frequency component lh3，hl3，hh3；
Step A2, enhancing the features reversely through a residual attention module, a wavelet inverse pooling layer and an attention feedback module; firstly, the low-frequency component ll of the third level3And a high frequency component lh3，hl3，hh3Obtaining transformed low-frequency components ll through residual attention modules respectively4And a high frequency component lh4，hl4hh4Inputting the transformed low-frequency component and high-frequency component into a wavelet anti-pooling layer for combined reconstruction, wherein the output characteristics of the wavelet anti-pooling layer are used as the input of an attention feedback module, and the output of the attention feedback module is a third level to a second level ll2Feedback weight w3 of vector to convert the low-frequency component ll of the second level2Multiplying the feedback weight w3 to perform correction to obtain a corrected low-frequency component ll of the second layer2′；
Step A3, correcting ll2' component and second-level high-frequency component lh2，hl2，hh2Transformed by the input residual attention moduleLow frequency component ll5And a high frequency component lh5，hl5，hh5(ii) a Inputting the transformed low-frequency component and high-frequency component into a wavelet anti-pooling layer for combined reconstruction, wherein the output characteristics of the wavelet anti-pooling layer are used as the input of an attention feedback module, and the output of the attention feedback module is the second level to the first level ll1Feedback weight w2 of vector, and low-frequency component ll of first level1Multiplying the feedback weight w2 to perform correction to obtain a corrected low-frequency component ll of the second layer1′；
Step A4, correcting ll1' component and first-level high-frequency component lh1，hl1，hh1Obtaining transformed low-frequency component ll by inputting residual attention module6And a high frequency component lh6，hl6，hh6(ii) a Inputting the transformed low-frequency component and high-frequency component into a wavelet anti-pooling layer for combined reconstruction, taking the output characteristics of the wavelet anti-pooling layer as the input of an attention feedback module, taking the output of the attention feedback module as the feedback weight w1 of the first level to the initial input characteristics of the network, and inputting the initial input characteristics of the network F1inMultiplying by weight w1 to obtain the image feature F1 after the current stage is enhancedoutNamely the output of the first-stage network;
in the above way, the output F2 of the second stage network is further obtainedoutAnd the output of the second stage network F3out。
8. The underwater image enhancement method based on the progressive feedback network as claimed in claim 3, wherein: step S23 specifically includes: designing the network of the supervised attention Module in step S21, the network input being the enhanced image features output by the first or second stage network F1outOr F2outAnd normalized underwater images
Figure FDA0003213130430000041
Taking the first module for supervision attention as an example, the input image features F1 are first inputoutStep size of 3x3 after passing through a convolution kernelObtaining a residual image for convolution of 1, and adding the residual image and the underwater image according to pixels to obtain a stage underwater enhanced image R1The obtained stage underwater enhanced image R1Calculating a loss with the normalized tag image G according to step S3; then, the stage underwater enhanced image R1R is obtained as an input to the convolutional layer with a convolution kernel of 3x3 and a step size of 11And obtaining a feature weight w4 through a sigmoid function, wherein the feature weight is used as a guide weight for the network input image feature of the supervision attention module. I.e. the originally input enhanced image feature F1outMultiplying by weight w4 to obtain corrected enhanced image feature F2inAs output from the network of supervising attention modules. The calculation formula is as follows:
Figure FDA0003213130430000051
Figure FDA0003213130430000052
enhanced image features F1 for first and second stage network outputout and F2outRespectively obtaining corrected characteristics F2 after passing through the network of the supervision attention modulein and F3inAnd underwater enhanced image R in the generation stage1 and R2。
9. The underwater image enhancement method based on the progressive feedback network as claimed in claim 1, wherein: the step S3 includes the following steps;
step S31: designing a network target loss function, wherein the network total target loss function is as follows:
Figure FDA0003213130430000054
wherein ,
Figure FDA0003213130430000055
and
Figure FDA0003213130430000056
are each L1Loss and grid loss, λ1 and λ2Is each loss balance coefficient, is a real number dot product operation; the specific calculation formula of each loss is as follows:
Figure FDA0003213130430000057
wherein R is the final output result of the designed underwater image enhancement network, R1 and R2Stage underwater enhanced images generated in the two monitoring attention modules are respectively, G is a normalized label image, | |1Is an absolute value operation;
Figure FDA0003213130430000053
wherein
Figure FDA0003213130430000061
Representing the Frobenius square norm, j representing the j-th layer, phi representing the feature extraction network, here using a pre-trained VGG16 network,
Figure FDA0003213130430000062
a Gram matrix representing the j-th layer activation characteristics extracted by the network phi is defined as follows:
Figure FDA0003213130430000063
where x represents the input image and phi represents the feature extraction network, here a pre-trained VGG16 network is usedj(x)h，w，cTo representThe j-th layer of activation features, phi, obtained from the input image x in the feature extraction network phij(x)h，w，c′Representing the transpose of the j-th layer of activation features obtained by the input image x in the feature extraction network phi, C, C' representing the number of feature channels, h representing the feature height, w representing the feature width, Cj×Hj×WjThe size of the features is activated for layer j.
10. The underwater image enhancement method based on the progressive feedback network as claimed in claim 1, wherein: the step S4 includes the following steps;
step S41: randomly dividing the matched underwater images and the label images into a plurality of batches, wherein each batch comprises N pairs of images;
step S42: inputting the underwater image into the image enhancement network in the step S2 to obtain a final enhanced image and a stage underwater enhanced image;
step S43: calculating the gradient of each parameter in the image enhancement network by using a back propagation method according to the total target loss function of the image enhancement network, and updating the parameter of the image enhancement network by using a random gradient descent method;
step S44: and (4) repeating the steps from S41 to S43 image enhancement network training by taking batches as units until the target loss function value of the image enhancement network converges to Nash balance, and storing the network parameters to finish the training process of the image enhancement network.
CN202110935907.7A 2021-08-16 2021-08-16 Underwater image enhancement method based on progressive feedback network Active CN113658072B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110935907.7A CN113658072B (en) 2021-08-16 2021-08-16 Underwater image enhancement method based on progressive feedback network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110935907.7A CN113658072B (en) 2021-08-16 2021-08-16 Underwater image enhancement method based on progressive feedback network

Publications (2)

Publication Number Publication Date
CN113658072A true CN113658072A (en) 2021-11-16
CN113658072B CN113658072B (en) 2023-08-08

Family

ID=78480347

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110935907.7A Active CN113658072B (en) 2021-08-16 2021-08-16 Underwater image enhancement method based on progressive feedback network

Country Status (1)

Country Link
CN (1) CN113658072B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018227105A1 (en) * 2017-06-08 2018-12-13 The United States Of America, As Represented By The Secretary, Department Of Health And Human Services Progressive and multi-path holistically nested networks for segmentation
CN112288658A (en) * 2020-11-23 2021-01-29 杭州师范大学 Underwater image enhancement method based on multi-residual joint learning
CN112581373A (en) * 2020-12-14 2021-03-30 北京理工大学 Image color correction method based on deep learning
CN112837232A (en) * 2021-01-13 2021-05-25 山东省科学院海洋仪器仪表研究所 Underwater image enhancement and detail recovery method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018227105A1 (en) * 2017-06-08 2018-12-13 The United States Of America, As Represented By The Secretary, Department Of Health And Human Services Progressive and multi-path holistically nested networks for segmentation
CN112288658A (en) * 2020-11-23 2021-01-29 杭州师范大学 Underwater image enhancement method based on multi-residual joint learning
CN112581373A (en) * 2020-12-14 2021-03-30 北京理工大学 Image color correction method based on deep learning
CN112837232A (en) * 2021-01-13 2021-05-25 山东省科学院海洋仪器仪表研究所 Underwater image enhancement and detail recovery method

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
张清博;张晓晖;韩宏伟;: "一种基于深度卷积神经网络的水下光电图像质量优化方法", 光学学报, no. 11 *
林森;刘世本;唐延东;: "多输入融合对抗网络的水下图像增强", 红外与激光工程, no. 05 *

Also Published As

Publication number Publication date
CN113658072B (en) 2023-08-08

Similar Documents

Publication Publication Date Title
CN112288658B (en) Underwater image enhancement method based on multi-residual joint learning
CN110599409B (en) Convolutional neural network image denoising method based on multi-scale convolutional groups and parallel
CN111275643B (en) Real noise blind denoising network system and method based on channel and space attention
CN111369487B (en) Hyperspectral and multispectral image fusion method, system and medium
CN107590779B (en) Image denoising and deblurring method based on image block clustering dictionary training
CN110276726B (en) Image deblurring method based on multichannel network prior information guidance
CN113362250B (en) Image denoising method and system based on dual-tree quaternary wavelet and deep learning
CN108765330B (en) Image denoising method and device based on global and local prior joint constraint
CN112767279B (en) Underwater image enhancement method for generating countermeasure network based on discrete wavelet integration
CN110533614B (en) Underwater image enhancement method combining frequency domain and airspace
CN113284061A (en) Underwater image enhancement method based on gradient network
CN114511480A (en) Underwater image enhancement method based on fractional order convolution neural network
CN117593235B (en) Retinex variation underwater image enhancement method and device based on depth CNN denoising prior
CN117911302A (en) Underwater low-illumination image enhancement method based on conditional diffusion model
CN117422653A (en) Low-light image enhancement method based on weight sharing and iterative data optimization
CN113658072B (en) Underwater image enhancement method based on progressive feedback network
CN114022371B (en) Defogging device and defogging method based on space and channel attention residual error network
Kumar et al. Image Deconvolution using Deep Learning-based Adam Optimizer
CN115601260A (en) Hyperspectral image restoration method driven by neural network and optimization model in combined mode
CN114862711A (en) Low-illumination image enhancement and denoising method based on dual complementary prior constraints
CN114926354A (en) Method, system and medium for decomposing underwater image restoration by using deep low-rank non-negative matrix
CN111325690A (en) Self-adaptive underwater image enhancement method based on differential evolution algorithm
CN113379638B (en) SAR image speckle removing method based on wavelet transformation and interval dense network
CN115731199A (en) Non-paired underwater image enhancement method based on wavelet transformation
CN114723614B (en) Image restoration method based on wavelet domain ADMM depth network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant