CN117291962B - Deblocking effect method of lightweight neural network based on channel decomposition - Google Patents
Deblocking effect method of lightweight neural network based on channel decomposition Download PDFInfo
- Publication number
- CN117291962B CN117291962B CN202311588102.5A CN202311588102A CN117291962B CN 117291962 B CN117291962 B CN 117291962B CN 202311588102 A CN202311588102 A CN 202311588102A CN 117291962 B CN117291962 B CN 117291962B
- Authority
- CN
- China
- Prior art keywords
- feature
- image
- characteristic
- enhancement
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 48
- 230000000694 effects Effects 0.000 title claims abstract description 35
- 238000000354 decomposition reaction Methods 0.000 title claims abstract description 20
- 230000006835 compression Effects 0.000 claims abstract description 10
- 238000007906 compression Methods 0.000 claims abstract description 10
- 230000002708 enhancing effect Effects 0.000 claims abstract description 7
- 239000000284 extract Substances 0.000 claims abstract description 5
- 238000000605 extraction Methods 0.000 claims description 18
- 238000012549 training Methods 0.000 claims description 15
- 230000004927 fusion Effects 0.000 claims description 9
- 238000010606 normalization Methods 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 7
- 230000004913 activation Effects 0.000 claims description 6
- 230000003044 adaptive effect Effects 0.000 claims description 6
- 230000015572 biosynthetic process Effects 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 abstract description 7
- 230000000903 blocking effect Effects 0.000 abstract description 4
- 230000007246 mechanism Effects 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 6
- 238000013139 quantization Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000000052 comparative effect Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000003062 neural network model Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000002238 attenuated effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/40—Analysis of texture
- G06T7/41—Analysis of texture based on statistical description of texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Probability & Statistics with Applications (AREA)
- Image Processing (AREA)
Abstract
The invention belongs to the field of image compression and quality enhancement, and particularly provides a deblocking method of a lightweight neural network based on channel decomposition, which is used for rapidly and efficiently removing the blocking effect of a compressed image and realizing the image quality enhancement. Firstly, dismantling a Y channel into a texture image Yt and a structural image Ys, so as to more perfectly reserve and utilize texture information, and designing a branching module to respectively extract characteristics of the Yt and YCbCr channels; then, designing a residual error module based on a channel attention mechanism and frequency domain convolution, and constructing a lightweight deblocking effect enhanced neural network in a stacking manner; and finally, enhancing the YCbCr image based on the lightweight deblocking effect enhancement neural network, and converting to obtain an enhanced RGB image. In summary, the lightweight deblocking enhanced neural network provided by the invention has the advantages of simple structure, extremely small parameter quantity and extremely small calculation complexity, and can ensure a good image deblocking restoration effect.
Description
Technical Field
The invention belongs to the field of image compression and quality enhancement, relates to an image deblocking method based on texture and structure decomposition, and particularly provides a lightweight neural network deblocking method based on channel decomposition.
Background
The image is one of main media for human to acquire information from the outside, and along with development of technology and rising of digital media, the digital image gradually plays an important role in the production and life of people. Image compression coding is an indispensable ring in digital image transmission, and is used to convert an image into a bit stream, and common compression methods include JPEG, JPEGXR, PNG, HEVC, and the like.
The main steps of image coding are color conversion, frequency domain transformation, quantization and coding, and image decoding corresponds to the inverse operation, wherein the key factors influencing the image quality are quantization, the larger the quantization step length is, the fewer bits are used, the larger the error is, the larger the difference between the decoded image and the original image is, and the worse the quality is; otherwise, the better the quality. In a practical application scenario, there are often limitations on transmission bandwidth and memory space, and it is necessary to control quantization step size to transmit and store image data using fewer bits, so that blocking effect of low quality images is caused, and human eye visual effect of decoded images is extremely poor. In order to balance the limitation of compression transmission and visual effect, the quality enhancement post-processing of image codec is important. In recent years, deep learning has shown surprising potential in low-level visual tasks such as denoising, deblocking effects, super-resolution and the like; with the updating of technology, more and more enhancement networks select larger parameter amounts and more construction modes of resource consumption to obtain better enhancement effects, and the higher requirements on the computing power and the video memory of the computer are raised.
Disclosure of Invention
The invention aims to provide a deblocking method of a lightweight neural network based on channel decomposition, which is used for rapidly and efficiently removing the blocking effect of a compressed image and realizing image quality enhancement.
In order to achieve the above purpose, the invention adopts the following technical scheme:
the technical scheme adopted by the invention is as follows:
a lightweight neural network deblocking method based on channel decomposition, comprising the steps of:
step 1, data preprocessing;
step 1.1, adopting JPEG compression as an image coder-decoder to convert and compress an original RGB image into YCbCr three-channel data, wherein Y channels represent brightness, and Cb channels and Cr channels represent chromaticity;
step 1.2. Decomposing the Y-channel image in the YCbCr three-channel data into a structural image Y by adopting a gray image structure and a texture decomposition method based on an optimized iterative loop s And texture image Y t And texture image Y t Combining with YCbCr three-channel data to form four-channel data [ Y ] t ,Y,Cb,Cr]；
Step 2, constructing a lightweight deblocking effect enhanced neural network and finishing training;
step 2.1. Constructing a lightweight deblocking enhanced neural network, comprising: the input of the lightweight deblocking effect enhanced neural network is four-channel data [ Y ] t ,Y,Cb,Cr]The output of the lightweight deblocking enhanced neural network is an enhanced YCbCr image;
training a lightweight deblocking enhancement neural network, namely setting training parameters by using MSE loss as a loss function, and training the lightweight deblocking enhancement neural network;
step 3, four-channel input data [ Y ] of the image to be processed t ,Y,Cb,Cr]The light-weight deblocking effect enhancement neural network is input to the trained light-weight deblocking effect enhancement neural network, the light-weight deblocking effect enhancement neural network outputs an enhanced YCbCr image, and then the enhanced YCbCr image is subjected to color inverse transformation to obtain a reconstructed RGB image.
Further, in step 1.2, the structural image Y s And texture image Y t The method comprises the following steps:
Y s =ExtractStructure(Y)，Y t =(Y-Y s )+mean(Y s )，
wherein extrastructure represents a gray image structure and texture decomposition method based on an optimization iteration loop, and mean represents a mean operation.
Further, in step 2.1, the feature extraction module extracts the branch H from the texture feature 1 And image feature extraction branch H 2 The texture feature extraction branch H 1 And image feature extraction branch H 2 Are each constituted by a convolution layer Conv3×3, texture image Y t Through texture feature extraction branch H 1 Extracting to obtain texture feature H 1out The method comprises the steps of carrying out a first treatment on the surface of the YCbCr three-channel data passes through image feature extraction branch H 2 Extracting to obtain image characteristic H 2out The method comprises the steps of carrying out a first treatment on the surface of the Texture feature H 1out And image feature H 2out Splice formation feature H 3in Inputting the characteristics to a characteristic fusion module;
the feature fusion module adopts a channel attention module H 3 The structure is as follows: adapteveavgpool2d+Conv1×1+ReLU+Conv1×1+Sigmoid, adapteveavgpool2d represents an adaptive average pooling layer, reLU represents a ReLU activation function, sigmoid represents a Sigmoid activation function, and the output of the feature fusion module is feature H 3out ；
The characteristic enhancement module is composed of a shallow characteristic enhancement module Q 1 Middle layer feature enhancement module Q 2 And deep feature enhancement module Q 3 The shallow characteristic enhancement module Q 1 Comprising an enhancement layer S 1 Enhancement layer S 2 And enhancement layer S 3 Enhancement layer S 1 Is characterized by H 3out Feature H 3out Obtaining feature S through SUnit 1mid Feature S 1mid And feature H 3out Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 1 Output characteristics S of (2) 1out The method comprises the steps of carrying out a first treatment on the surface of the Enhancement layer S 2 Is input as feature S 1out Feature S 1out Obtaining the characteristic S through SUnit+SUnit 2mid Feature S 2mid And features S 1out Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 2 Output characteristics S of (2) 2out The method comprises the steps of carrying out a first treatment on the surface of the Enhancement layer S 3 Is input as feature S 2out Feature S 2out Obtaining the characteristic S through SUnit+SUnit 3mid Feature S 3mid And features S 2out Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 3 Output characteristics S of (2) 3out The method comprises the steps of carrying out a first treatment on the surface of the Middle layer feature enhancement module Q 2 From 4 frequency domain residual modules R f Series connection structure, middle layer characteristic enhancing module Q 2 Is characterized by the input of (a)S 3out Middle layer feature enhancement module Q 2 The output of (a) is characteristic Q 2out The method comprises the steps of carrying out a first treatment on the surface of the Deep feature enhancement module Q 3 Comprising an enhancement layer S 4 Enhancement layer S 5 And enhancement layer S 6 The method comprises the steps of carrying out a first treatment on the surface of the Feature Q 2out And features S 3out Added as enhancement layer S 4 Input S of (2) 4in Obtaining the characteristic S through SUnit+SUnit 4mid Feature S 4mid And input S 4in Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 4 Output characteristics S of (2) 4out The method comprises the steps of carrying out a first treatment on the surface of the Feature S 4out And features S 2out Added as enhancement layer S 5 Input S of (2) 5in Obtaining the characteristic S through SUnit+SUnit 5mid Feature S 5mid And input S 5in Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 5 Output characteristics S of (2) 5out The method comprises the steps of carrying out a first treatment on the surface of the Feature S 5out And features S 1out Added as enhancement layer S 6 Input S of (2) 6in Obtaining the characteristic S through SUnit+SUnit 6mid Feature S 6mid And input S 6in Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 6 Output characteristics S of (2) 6out The method comprises the steps of carrying out a first treatment on the surface of the SUnit represents a feature enhancement unit;
the image reconstruction module consists of a convolution layer Conv3×3, and the input of the image reconstruction module is the characteristic S 6out The output is an enhanced YCbCr image.
Further, the characteristic enhancing unit SUnit is formed by a frequency domain residual error module R f And channel residual error module R c In a series configuration, wherein,
frequency domain residual error module R f Comprising airspace branch B 1 And frequency domain branch B 2 Space domain branch B 1 The structure of (2) is as follows: conv1×1+Conv3×3+ReLU+Conv1×1, frequency domain branch B 2 The structure of (2) is as follows: FFT+Conv1+ReLU+Conv1+iFFT, FFT means fast Fourier transform, iFFT means inverse fast Fourier transform; frequency domain residual error module R f Input R of (2) fin Respectively input to the airspace branch B after the first LayerNorm normalization 1 And frequency domain branch B 2 Space domain branch B 1 And frequency domain branch B 2 Is passed through the residueAfter the difference connection, the characteristic R is obtained after the second LayerNorm normalization and the Conv1×1 convolution layer fmid Characteristic R fmid And input R fin Module R for obtaining frequency domain residual error by adding f Output R of (2) fout ；
Channel residual module R c In, input R cin Obtaining the characteristic R through LayerNorm+Conv1×1+Conv3×3+ReLU cmid1 Adopting adaptive AvgPool2d+Conv1X1 to form a simplified version channel attention structure, and calculating to obtain a characteristic R cmid1 Feature weights of (a)wFeatures R cmid1 And feature weightwMultiplying to obtain feature R cmid2 Characteristic R cmid2 The characteristic obtained by the Conv1×1 of the convolution layer is added with the input Rcin to obtain a characteristic R cmid3 Characteristic R cmid3 Obtaining the characteristic R through LayerNorm+Conv1+ReLU+Conv1+1 cmid4 Characteristic R cmid4 And features R cmid3 Module R for obtaining channel residual error by adding c Output R of (2) cout 。
Based on the technical scheme, the invention has the beneficial effects that:
the invention provides a deblocking effect method of a lightweight neural network based on channel decomposition, which comprises the steps of firstly, disassembling a Y channel into a texture image Yt and a structural image Ys, so as to more perfectly reserve and utilize texture information, and designing a branching module to respectively extract characteristics of the Yt and the YCbCr channels; then, designing a residual error module based on a channel attention mechanism and frequency domain convolution, and ensuring that the model acquires and restores image information in a channel dimension and a frequency domain dimension; the lightweight deblocking effect enhanced neural network is constructed by stacking lightweight residual blocks, the parameter quantity and the calculation complexity are controlled in a minimum range, and the calculation cost is effectively saved; and finally, enhancing the YCbCr image based on the lightweight deblocking effect enhancement neural network, and converting to obtain an enhanced RGB image. In summary, the lightweight deblocking enhanced neural network provided by the invention has the advantages of simple structure, extremely small parameter quantity and extremely small calculation complexity, and can ensure a good image deblocking restoration effect.
Drawings
Fig. 1 is a flow chart of a deblocking method for lightweight neural networks based on channel decomposition in the present invention.
FIG. 2 is a schematic diagram of a lightweight deblocking enhanced neural network according to the present invention.
FIG. 3 is a schematic diagram of a feature enhancement module of a lightweight deblocking enhanced neural network according to the present invention.
FIG. 4 is a frequency domain residual module R of the characteristic enhancement module of the present invention f Is a schematic structural diagram of the (c).
FIG. 5 shows a channel residual module R of the feature enhancement module of the present invention c Is a schematic structural diagram of the (c).
FIG. 6 is a Y-channel image and a structural image Y according to the present invention s Texture image Y t Is shown in the drawings.
Detailed Description
In order to make the objects, technical solutions and advantageous effects of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings and examples.
The present embodiment provides a deblocking method for lightweight neural networks based on texture and structure decomposition, which is used for rapidly and efficiently removing the blocking effect of compressed images, and the flow is shown in fig. 1, and specifically includes the following steps:
step 1, data preprocessing;
adopting JPEG compression as an image coder-decoder, under the set compression quality, obtaining a binary data stream by carrying out color transformation, frequency domain transformation and quantization on an original RGB image, and obtaining a compressed YCbCr image by carrying out inverse quantization and frequency domain inverse transformation on a decoding end, namely converting and compressing the original RGB image into YCbCr three-channel data, wherein Y channels represent brightness, and Cb channels and Cr channels represent chromaticity;
step 1.2. Decomposing the Y-channel image in the YCbCr three-channel data into a structural image Y by adopting a gray image structure and texture decomposition method (expressed as: extractStructure) based on an optimization iteration loop s And texture image Y t And texture image Y t Combining with YCbCr three channel data to form four channel data, denoted as [ Y ] t ,Y,Cb,Cr]；
Structural image Y s And texture image Y t The method comprises the following steps:
Y s =ExtractStructure(Y)，Y t =(Y-Y s )+mean(Y s )，
wherein mean represents the operation of solving the mean;
the gray image structure and texture decomposition method based on the optimized iterative loop is a well-known technology in the field, the decomposition result is schematically shown in fig. 6, and the decomposition result is a Y-channel image and a structural image Y sequentially from left to right s And texture image Y t ；
Step 2, constructing a lightweight deblocking effect enhanced neural network and finishing training;
step 2.1, constructing a lightweight deblocking effect enhanced neural network;
the lightweight deblocking enhanced neural network is shown in fig. 2, and includes: the input of the lightweight deblocking effect enhanced neural network is four-channel data [ Y ] t ,Y,Cb,Cr]The output of the lightweight deblocking enhanced neural network is an enhanced YCbCr image;
the characteristic extraction module extracts a branch H from texture characteristics 1 And image feature extraction branch H 2 The texture feature extraction branch H 1 And image feature extraction branch H 2 Are each constituted by a convolution layer Conv3×3, texture image Y t Through texture feature extraction branch H 1 Extracting to obtain texture feature H 1out The dimension is 16; YCbCr three-channel data passes through image feature extraction branch H 2 Extracting to obtain image characteristic H 2out Dimension 48; texture feature H 1out And image feature H 2out Splice formation feature H 3in Inputting the characteristics to a characteristic fusion module;
the characteristic fusion module adopts a channel attention module H 3 The structure is as follows: adapteveavgpool2d+Conv1×1+ReLU+Conv1×1+Sigmoid, adapteveavgpool2d represents an adaptive average pooling layer, reLU represents a ReLU activation function, sigmoid represents a Sigmoid activation function; the output of the feature fusion module is the feature H 3out Dimension 64;
the characteristic enhancement module is composed of a shallow characteristic enhancement module Q 1 Middle layer feature enhancement module Q 2 And deep feature enhancement module Q 3 The constitution is as shown in fig. 3; the feature enhancement module comprises a plurality of feature enhancement units SUnit, and the feature dimension is kept to be 64;
the characteristic enhancement unit SUnit is formed by a frequency domain residual error module R f And channel residual error module R c Series connection is formed, the frequency domain residual error module R f As shown in fig. 4, the design is initially to expand the receptive field under convolution kernels of a defined size and to perform calculations and enhancements quickly and efficiently, thus FFT fast fourier frequency domain convolution is employed to obtain more powerful feature enhancement capability; frequency domain residual error module R f Comprising airspace branch B 1 And frequency domain branch B 2 Space domain branch B 1 The structure of (2) is as follows: conv1×1+Conv3×3+ReLU+Conv1×1, frequency domain branch B 2 The structure of (2) is as follows: FFT+Conv1+ReLU+Conv1+iFFT, FFT means fast Fourier transform, iFFT means inverse fast Fourier transform; frequency domain residual error module R f Input R of (2) fin Respectively input to B after first LayerNorm normalization 1 Branch and B 2 Branch, B 1 Branch and B 2 The output of the branch is subjected to residual connection and then is subjected to LayerNorm normalization for the second time, and is subjected to Conv1×1 of a convolution layer to obtain a characteristic R fmid Characteristic R fmid And input R fin Module R for obtaining frequency domain residual error by adding f Output R of (2) fout The method comprises the steps of carrying out a first treatment on the surface of the The channel residual error module R c As shown in fig. 5, the purpose of the design is to complement the frequency domain residual block R f The attention of the channel layer information is lost, so a channel attention mechanism is adopted; channel residual module R c Input R of (2) cin Obtaining the characteristic R through LayerNorm+Conv1×1+Conv3×3+ReLU cmid1 Adopting adaptive AvgPool2d+Conv1X1 to form a simplified version channel attention structure, and calculating to obtain a characteristic R cmid1 Feature weights of (a)wFeatures R cmid1 And feature weightwMultiplication to obtainFeature R cmid2 Characteristic R cmid2 The characteristic after the Conv1×1 of the convolution layer is added with the input Rcin to obtain a characteristic R cmid3 Then obtaining the characteristic R through LayerNorm+Conv1×1+ReLU+Conv1×1 cmid4 Characteristic R cmid4 And features R cmid3 Module R for obtaining channel residual error by adding c Output R of (2) cout ；
The shallow characteristic enhancement module Q 1 Comprising an enhancement layer S 1 Enhancement layer S 2 And enhancement layer S 3 Enhancement layer S 1 Is characterized by H 3out Feature H 3out Obtaining feature S through SUnit 1mid Feature S 1mid And feature H 3out Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 1 The output of (i.e. feature S) 1out The method comprises the steps of carrying out a first treatment on the surface of the Enhancement layer S 2 Is input as feature S 1out Feature S 1out Obtaining the characteristic S through SUnit+SUnit 2mid Feature S 2mid And features S 1out Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 2 The output of (i.e. feature S) 2out The method comprises the steps of carrying out a first treatment on the surface of the Enhancement layer S 3 Is input as feature S 2out Feature S 2out Obtaining the characteristic S through SUnit+SUnit 3mid Feature S 3mid And features S 2out Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 3 The output of (i.e. feature S) 3out ；
The middle layer characteristic enhancing module Q 2 From 4 frequency domain residual modules R f Series connection structure, middle layer characteristic enhancing module Q 2 Is input as feature S 3out Middle layer feature enhancement module Q 2 The output of (a) is characteristic Q 2out ；
The deep feature enhancement module Q 3 Comprising an enhancement layer S 4 Enhancement layer S 5 And enhancement layer S 6 The method comprises the steps of carrying out a first treatment on the surface of the Feature Q 2out And features S 3out Added as enhancement layer S 4 Input S of (2) 4in Obtaining the characteristic S through SUnit+SUnit 4mid Feature S 4mid And input S 4in Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 4 The output of (i.e. feature S) 4out The method comprises the steps of carrying out a first treatment on the surface of the Special purposeSign S 4out And features S 2out Added as enhancement layer S 5 Input S of (2) 5in Obtaining the characteristic S through SUnit+SUnit 5mid Feature S 5mid And input S 5in Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 5 The output of (i.e. feature S) 5out The method comprises the steps of carrying out a first treatment on the surface of the Feature S 5out And features S 1out Added as enhancement layer S 6 Input S of (2) 6in Obtaining the characteristic S through SUnit+SUnit 6mid Feature S 6mid And input S 6in Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 6 The output of (i.e. feature S) 6out ；
The image reconstruction module consists of a convolution layer Conv3×3, and the input of the image reconstruction module is the characteristic S 6out Outputting an enhanced YCbCr image;
conv3×3 represents a convolution layer having a convolution kernel size of 3×3, and Conv1×1 is the same; the "+" sign indicates a sequential connection;
step 2.2, training a lightweight deblocking effect enhanced neural network;
setting training parameters, initializing model weights, training a lightweight deblocking effect enhanced neural network by adopting an AdamW optimizer, adjusting the optimizer in a gradual attenuation learning rate mode, wherein the initial learning rate lr is 5e-4, the training learning rate is attenuated by 50% every 20 times, and a loss function is selected as MSE loss, specifically comprising:
，
wherein Y is a reconstructed RGB image, X is an original RGB image, (i, j) represents pixel coordinates, M, N is the length and width of the image respectively;
step 3, four-channel input data [ Y ] of the image to be processed t ,Y,Cb,Cr]The light-weight deblocking effect enhancement neural network is input to the trained light-weight deblocking effect enhancement neural network, the light-weight deblocking effect enhancement neural network outputs an enhanced YCbCr image, and then the enhanced YCbCr image is subjected to color inverse transformation to obtain a reconstructed RGB image.
The beneficial effects of the invention are verified by combining simulation tests:
in the embodiment, the DIV2K data set is used as a training set to train the lightweight deblocking effect enhancement neural network, LIVE1 and BSDS500 image sets are used as test sets to verify the enhancement effect of the model, and the PSNR value is used as a judgment standard for the enhancement effect. In the present invention, the compressed RGB image is usually directly enhanced in the conventional deblocking method, and the common enhancement model includes RNAN, FBCNN and NAFNet networks, which are taken as comparative examples, and the enhancement effects of the present invention and the comparative examples are shown in table 1:
TABLE 1
As can be seen from table 1, the restoration effect of the present invention is excellent for compressed images having compression qualities of 10, 20, 30, and 40.
In addition, the parameter number and the flow of the neural network model represent the size and the calculation complexity of the model respectively, and the smaller the parameter number is, the smaller the flow is, the less calculation resources are consumed by the neural network model, and the faster the running speed is; the parameters and flows of the invention and comparative examples are shown in table 2:
TABLE 2
Wherein, the flow is statistically calculated under input data of size 56×56; as can be seen from table 2, the parameters and the flows values of the invention are smaller, so that the training and running cost of the neural network model is effectively reduced; in particular, compared to RNAN, the parameter of the present invention is reduced by 89.84% and Flots is reduced by 90.70%.
While the invention has been described in terms of specific embodiments, any feature disclosed in this specification may be replaced by alternative features serving the equivalent or similar purpose, unless expressly stated otherwise; all of the features disclosed, or all of the steps in a method or process, except for mutually exclusive features and/or steps, may be combined in any manner.
Claims (2)
1. A method for deblocking a lightweight neural network based on channel decomposition, comprising the steps of:
step 1, data preprocessing;
step 1.1, adopting JPEG compression as an image coder-decoder to convert and compress an original RGB image into YCbCr three-channel data, wherein Y channels represent brightness, and Cb channels and Cr channels represent chromaticity;
step 1.2. Decomposing the Y-channel image in the YCbCr three-channel data into a structural image Y by adopting a gray image structure and a texture decomposition method based on an optimized iterative loop s And texture image Y t And texture image Y t Combining with YCbCr three-channel data to form four-channel data [ Y ] t ,Y,Cb,Cr]；
Step 2, constructing a lightweight deblocking effect enhanced neural network and finishing training;
step 2.1. Constructing a lightweight deblocking enhanced neural network, comprising: the input of the lightweight deblocking effect enhanced neural network is four-channel data [ Y ] t ,Y,Cb,Cr]The output of the lightweight deblocking enhanced neural network is an enhanced YCbCr image;
the feature extraction module extracts the branch H from the texture features 1 And image feature extraction branch H 2 The texture feature extraction branch H 1 And image feature extraction branch H 2 Are each constituted by a convolution layer Conv3×3, texture image Y t Through texture feature extraction branch H 1 Extracting to obtain texture feature H 1out The method comprises the steps of carrying out a first treatment on the surface of the YCbCr three-channel data passes through image feature extraction branch H 2 Extracting to obtain image characteristic H 2out The method comprises the steps of carrying out a first treatment on the surface of the Texture feature H 1out And image feature H 2out Splice formation feature H 3in Inputting the characteristics to a characteristic fusion module;
the feature fusion module adopts a channel attention module H 3 The structure is as follows: adapteveavgPool2d+Conv1×1+ReLU+Conv1×1+Sigmoid, adapteveavgPool2d represents an adaptive average pooling layer, reLU represents a ReLU activation function, sigmoid represents a Sigmoid activation function, conv1×1 represents a convolution layer with a convolution kernel size of 1×1, and the output of the feature fusion module is feature H 3out ；
The characteristic enhancement module is composed of a shallow characteristic enhancement module Q 1 Middle layer feature enhancement module Q 2 And deep feature enhancement module Q 3 The shallow characteristic enhancement module Q 1 Comprising an enhancement layer S 1 Enhancement layer S 2 And enhancement layer S 3 Enhancement layer S 1 Is characterized by H 3out Feature H 3out Obtaining feature S through SUnit 1mid Feature S 1mid And feature H 3out Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 1 Output characteristics S of (2) 1out The method comprises the steps of carrying out a first treatment on the surface of the Enhancement layer S 2 Is input as feature S 1out Feature S 1out Obtaining the characteristic S through SUnit+SUnit 2mid Feature S 2mid And features S 1out Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 2 Output characteristics S of (2) 2out The method comprises the steps of carrying out a first treatment on the surface of the Enhancement layer S 3 Is input as feature S 2out Feature S 2out Obtaining the characteristic S through SUnit+SUnit 3mid Feature S 3mid And features S 2out Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 3 Output characteristics S of (2) 3out The method comprises the steps of carrying out a first treatment on the surface of the Middle layer feature enhancement module Q 2 From 4 frequency domain residual modules R f Series connection structure, middle layer characteristic enhancing module Q 2 Is input as feature S 3out Middle layer feature enhancement module Q 2 The output of (a) is characteristic Q 2out The method comprises the steps of carrying out a first treatment on the surface of the Deep feature enhancement module Q 3 Comprising an enhancement layer S 4 Enhancement layer S 5 And enhancement layer S 6 The method comprises the steps of carrying out a first treatment on the surface of the Feature Q 2out And features S 3out Added as enhancement layer S 4 Input S of (2) 4in Obtaining the characteristic S through SUnit+SUnit 4mid Feature S 4mid And input S 4in Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 4 Output characteristics S of (2) 4out The method comprises the steps of carrying out a first treatment on the surface of the Feature S 4out And features S 2out Added as enhancement layer S 5 Is the input of (2)S in 5in Obtaining the characteristic S through SUnit+SUnit 5mid Feature S 5mid And input S 5in Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 5 Output characteristics S of (2) 5out The method comprises the steps of carrying out a first treatment on the surface of the Feature S 5out And features S 1out Added as enhancement layer S 6 Input S of (2) 6in Obtaining the characteristic S through SUnit+SUnit 6mid Feature S 6mid And input S 6in Added and then subjected to Conv1×1+ReLU to obtain an enhancement layer S 6 Output characteristics S of (2) 6out The method comprises the steps of carrying out a first treatment on the surface of the SUnit represents a feature enhancement unit;
the image reconstruction module consists of a convolution layer Conv3×3, and the input of the image reconstruction module is the characteristic S 6out Outputting an enhanced YCbCr image;
the characteristic enhancement unit SUnit is formed by a frequency domain residual error module R f And channel residual error module R c In a series configuration, wherein,
frequency domain residual error module R f Comprising airspace branch B 1 And frequency domain branch B 2 Space domain branch B 1 The structure of (2) is as follows: conv1×1+Conv3×3+ReLU+Conv1×1, conv3×3 representing a convolution layer with a convolution kernel size of 3×3; frequency domain branch B 2 The structure of (2) is as follows: FFT+Conv1+ReLU+Conv1+iFFT, FFT means fast Fourier transform, iFFT means inverse fast Fourier transform; frequency domain residual error module R f Input R of (2) fin Respectively input to the airspace branch B after the first LayerNorm normalization 1 And frequency domain branch B 2 Space domain branch B 1 And frequency domain branch B 2 The output is subjected to residual connection and then is subjected to LayerNorm normalization for the second time, and is subjected to Conv1×1 of a convolution layer to obtain a characteristic R fmid Characteristic R fmid And input R fin Module R for obtaining frequency domain residual error by adding f Output R of (2) fout ；
Channel residual module R c In, input R cin Obtaining the characteristic R through LayerNorm+Conv1×1+Conv3×3+ReLU cmid1 LayerNorm means LayerNorm normalization; adopting adaptive AvgPool2d+Conv1X1 to form a simplified version channel attention structure, and calculating to obtain a characteristic R cmid1 Feature weights of (a)wWill be speciallySign R cmid1 And feature weightwMultiplying to obtain feature R cmid2 Characteristic R cmid2 The characteristic obtained by the Conv1×1 of the convolution layer is added with the input Rcin to obtain a characteristic R cmid3 Characteristic R cmid3 Then obtaining the characteristic R through LayerNorm+Conv1×1+ReLU+Conv1×1 cmid4 Characteristic R cmid4 And features R cmid3 Module R for obtaining channel residual error by adding c Output R of (2) cout ；
Training a lightweight deblocking enhancement neural network, namely setting training parameters by using MSE loss as a loss function, and training the lightweight deblocking enhancement neural network;
step 3, four-channel input data [ Y ] of the image to be processed t ,Y,Cb,Cr]The light-weight deblocking effect enhancement neural network is input to the trained light-weight deblocking effect enhancement neural network, the light-weight deblocking effect enhancement neural network outputs an enhanced YCbCr image, and then the enhanced YCbCr image is subjected to color inverse transformation to obtain a reconstructed RGB image.
2. The method for deblocking of channel decomposition-based lightweight neural networks of claim 1, wherein in step 1.2, the structural image Y s And texture image Y t The method comprises the following steps:
Y s =ExtractStructure(Y)，Y t =(Y-Y s )+mean(Y s )，
wherein extrastructure represents a gray image structure and texture decomposition method based on an optimization iteration loop, and mean represents a mean operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311588102.5A CN117291962B (en) | 2023-11-27 | 2023-11-27 | Deblocking effect method of lightweight neural network based on channel decomposition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311588102.5A CN117291962B (en) | 2023-11-27 | 2023-11-27 | Deblocking effect method of lightweight neural network based on channel decomposition |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117291962A CN117291962A (en) | 2023-12-26 |
| CN117291962B true CN117291962B (en) | 2024-02-02 |
Family
ID=89252153
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311588102.5A Active CN117291962B (en) | 2023-11-27 | 2023-11-27 | Deblocking effect method of lightweight neural network based on channel decomposition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117291962B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110120019A (en) * | 2019-04-26 | 2019-08-13 | 电子科技大学 | A kind of residual error neural network and image deblocking effect method based on feature enhancing |
| CN112053408A (en) * | 2020-09-04 | 2020-12-08 | 清华大学 | Face image compression method and device based on deep learning |
| CN112215767A (en) * | 2020-09-28 | 2021-01-12 | 电子科技大学 | Anti-blocking effect image video enhancement method |
| CN113362225A (en) * | 2021-06-03 | 2021-09-07 | 太原科技大学 | Multi-description compressed image enhancement method based on residual recursive compensation and feature fusion |
| WO2021208247A1 (en) * | 2020-04-17 | 2021-10-21 | 北京大学 | Mimic compression method and apparatus for video image, and storage medium and terminal |
| CN113962882A (en) * | 2021-09-29 | 2022-01-21 | 西安交通大学 | JPEG image compression artifact eliminating method based on controllable pyramid wavelet network |
| WO2023092813A1 (en) * | 2021-11-25 | 2023-06-01 | 苏州大学 | Swin-transformer image denoising method and system based on channel attention |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11457244B2 (en) * | 2018-04-09 | 2022-09-27 | Nokia Technologies Oy | Apparatus, a method and a computer program for video coding and decoding |
| EP4094443A1 (en) * | 2020-05-15 | 2022-11-30 | Huawei Technologies Co., Ltd. | Global skip connection based cnn filter for image and video coding |
-
2023
- 2023-11-27 CN CN202311588102.5A patent/CN117291962B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110120019A (en) * | 2019-04-26 | 2019-08-13 | 电子科技大学 | A kind of residual error neural network and image deblocking effect method based on feature enhancing |
| WO2021208247A1 (en) * | 2020-04-17 | 2021-10-21 | 北京大学 | Mimic compression method and apparatus for video image, and storage medium and terminal |
| CN112053408A (en) * | 2020-09-04 | 2020-12-08 | 清华大学 | Face image compression method and device based on deep learning |
| CN112215767A (en) * | 2020-09-28 | 2021-01-12 | 电子科技大学 | Anti-blocking effect image video enhancement method |
| CN113362225A (en) * | 2021-06-03 | 2021-09-07 | 太原科技大学 | Multi-description compressed image enhancement method based on residual recursive compensation and feature fusion |
| CN113962882A (en) * | 2021-09-29 | 2022-01-21 | 西安交通大学 | JPEG image compression artifact eliminating method based on controllable pyramid wavelet network |
| WO2023092813A1 (en) * | 2021-11-25 | 2023-06-01 | 苏州大学 | Swin-transformer image denoising method and system based on channel attention |
Non-Patent Citations (3)
| Title |
|---|
| Image deblocking via shape-adaptive low-rank prior and sparsity-based detail enhancement;Jing Hu 等;《Signal Processing: Image Communication》;1-11 * |
| SPIHT算法对BP神经网络图像压缩处理的改善;熊校良 等;《电子测量与仪器学报》;第22卷(第6期);7-11 * |
| 基于双路增强残差块连接的图像去块效应网络;冯洁丽 等;《无线电工程》;第52卷(第12期);2271-2279 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117291962A (en) | 2023-12-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110120019B (en) | Residual error neural network based on feature enhancement and image deblocking method | |
| CN111669587B (en) | Mimic compression method and device of video image, storage medium and terminal | |
| Dong et al. | Semantic communication system based on semantic slice models propagation | |
| CN110751597B (en) | Video super-resolution method based on coding damage repair | |
| Sun et al. | Reduction of JPEG compression artifacts based on DCT coefficients prediction | |
| Hu et al. | An image compression and encryption scheme based on deep learning | |
| CN108921910A (en) | The method of JPEG coding compression image restoration based on scalable convolutional neural networks | |
| CN114581544A (en) | Image compression method, computer device and computer storage medium | |
| CN110753225A (en) | Video compression method and device and terminal equipment | |
| Li et al. | Multiple description coding based on convolutional auto-encoder | |
| CN116563108A (en) | Residual error-based multispectral channel attention network underwater image super-resolution method | |
| CN112509071B (en) | Chroma information compression and reconstruction method assisted by luminance information | |
| Zhang et al. | A blind watermarking system based on deep learning model | |
| CN117291962B (en) | Deblocking effect method of lightweight neural network based on channel decomposition | |
| CN113938254A (en) | Attention mechanism-based layered source-channel joint coding transmission system and transmission method thereof | |
| CN110378466B (en) | Neural network difference-based quantization method and system | |
| CN110956669A (en) | Image compression coding method and system | |
| CN107146259A (en) | A kind of New Image compression method based on compressive sensing theory | |
| CN116843559A (en) | Underwater image enhancement method based on image processing and deep learning | |
| CN114596223A (en) | Quantization parameter self-adaptive convolution neural network loop filter and construction method thereof | |
| Yang et al. | Graph-convolution network for image compression | |
| CN115601242B (en) | Lightweight image super-resolution reconstruction method suitable for hardware deployment | |
| CN115358954B (en) | Attention-guided feature compression method | |
| CN110766766B (en) | Self-adaptive compressed sensing image coding method driven by sensing data | |
| El Zorkany | A Hybrid Image Compression Technique Using Neural Network and Vector Quantization With DCT. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |