基于双流多路径递归残差网络的压缩伪影去除方法Compression artifact removal method based on dual-stream multi-path recursive residual network
技术领域Technical field
本发明涉及一种图像和视频等媒体文件在存储和传输过程中压缩及去除伪影的方法。The invention relates to a method for compressing and removing artifacts during storage and transmission of media files such as images and videos.
背景技术Background technique
图像和视频等媒体文件在存储和传输过程中通常采用压缩算法来降低成本。基于信息论角度,图像压缩算法可分为有损压缩(如JPEG)和无损压缩(如PNG)两大类,与无损压缩相比,有损压缩具有更高的压缩比。但是,有损压缩往往会引起不可逆的信息丢失并导致压缩伪影,例如振铃、块效应和模糊,特别是对于低比特率的编码。这些压缩伪影不仅降低了用户体验,而且对许多初级图像处理任务造成不良的影响。JPEG是目前最广泛使用的图像压缩标准,通过采用基于块的离散余弦变换(BDCT,Block-based Discrete Cosine Transform)与粗略量化的策略,旨在减少像素间统计冗余信息,实现高压缩比。然而,在不考虑相邻块间的相关性时,容易出现块边界处的强度不连续。此外,高频离散余弦变换系数的截断也会导致振铃和模糊伪影。因此,有损压缩引起的压缩伪影通常可以认为是一种混合伪影。Media files such as images and videos usually use compression algorithms to reduce costs during storage and transmission. From the perspective of information theory, image compression algorithms can be divided into lossy compression (such as JPEG) and lossless compression (such as PNG). Compared with lossless compression, lossy compression has a higher compression ratio. However, lossy compression often causes irreversible information loss and compression artifacts, such as ringing, blocking, and blur, especially for low-bit-rate encoding. These compression artifacts not only degrade the user experience, but also adversely affect many primary image processing tasks. JPEG is currently the most widely used image compression standard. Through the use of block-based discrete cosine transform (BDCT, Block-based Discrete Cosine Transform) and coarse quantization strategies, it aims to reduce statistical redundancy between pixels and achieve a high compression ratio. However, when the correlation between adjacent blocks is not considered, the intensity discontinuity at the block boundary is prone to appear. In addition, the truncation of high-frequency discrete cosine transform coefficients can also cause ringing and blurring artifacts. Therefore, compression artifacts caused by lossy compression can generally be considered as hybrid artifacts.
早期的压缩伪影去除(CAR,Compression Artifacts Reduction)主要是基于滤波的方法,通过手动设计一个在块边界上工作的滤波器以减少块效应。此外,还有采用小波变换,自适应离散余弦变换(SA-DCT,Shape-Adaptive Discrete Cosine Transform),以及基于稀疏编码的方法等。利用前面提及的方法从压缩图像内获得无伪影图像时,通常会伴随有噪声边缘和不自然平滑区域。除了传统的压缩伪影去除方法,基于深度学***滑。另一方面,虽然深度网络可以提供更好的性能,但参数太多,网络难以训练并且深度网络模型的存储成本较传统方法也较 大。The early compression artifact removal (CAR, Compression Artifacts Reduction) was mainly based on filtering methods, which reduced the blocking effect by manually designing a filter that worked on the block boundary. In addition, there are also wavelet transform, adaptive discrete cosine transform (SA-DCT, Shape-Adaptive Discrete Cosine Transform), and methods based on sparse coding. When using the aforementioned method to obtain an artifact-free image from a compressed image, it is usually accompanied by noise edges and unnatural smooth areas. In addition to traditional compression artifact removal methods, methods based on deep learning obtain the best results by learning the nonlinear mapping relationship between the compressed image and the original image. However, due to the limitation of the shallow network, the reconstructed image appears excessively smooth. On the other hand, although the deep network can provide better performance, there are too many parameters, the network is difficult to train and the storage cost of the deep network model is higher than that of traditional methods.
此外,先前的工作或者集中于去除压缩伪影中一种具体的伪影,或者仅仅依赖于端到端的网络无针对地减少所有类型的伪影,但这样可能造成一种伪影的减少却无意增大另一种伪影的影响(例如,去除块效应的同时可能加剧模糊效应的影响)。一般地说,CAR可以分成去块效应,去振铃伪影和去模糊三个子任务。对于去块效应、去振铃伪影的任务,需要抑制干扰的高频信息,而对于去模糊任务,需要增强尖锐的边缘信息以及有用的高频信息。In addition, previous work either focused on removing a specific artifact in compression artifacts, or only relied on an end-to-end network to reduce all types of artifacts inadvertently, but this may result in a reduction of artifacts without intention. Increase the influence of another kind of artifact (for example, removing the blocking effect may exacerbate the influence of the blur effect). Generally speaking, CAR can be divided into three subtasks: de-blocking, de-ringing artifacts and de-blurring. For the tasks of deblocking and ringing artifacts, it is necessary to suppress the interference high-frequency information, and for the deblurring tasks, it is necessary to enhance the sharp edge information and useful high-frequency information.
由于去块效应、去振铃伪影与锐化边缘(去模糊)任务之间相互抵消,如果不区分这些伪影时,端到端的卷积网络在减少一种类型伪影的同时会增强其他类型的伪影。Since the tasks of deblocking, ringing artifacts, and edge sharpening (deblurring) cancel each other out, if these artifacts are not distinguished, the end-to-end convolutional network will reduce one type of artifact while enhancing the others. Type of artifacts.
发明内容Summary of the invention
本发明的目的是提供一种能够显著地抑制压缩伪影,减少失真,进一步增强重建图像的基于双流多路径递归残差网络的压缩伪影去除方法。The object of the present invention is to provide a compression artifact removal method based on a dual-stream multi-path recursive residual network that can significantly suppress compression artifacts, reduce distortion, and further enhance reconstructed images.
本发明是这样来实现上述目的的:The present invention achieves the above-mentioned objects in this way:
基于双流多路径递归残差网络的压缩伪影去除方法,基于双流多路径递归残差网络架构,利用图像高频(HF,high frequency)和低频(LF,low frequency)分量来针对性的去除压缩伪影。首先,网络将压缩失真图像分解为纹理层(包含HF分量)和结构层(包含LF分量);其次,利用多路径递归残差网络分别增强纹理和结构信息;最终,将结构和纹理信息合并后并馈送到回归网络中以生成最终的重建图像。Compression artifact removal method based on dual-stream multi-path recursive residual network, based on dual-stream multi-path recursive residual network architecture, using image high frequency (HF, high frequency) and low frequency (LF, low frequency) components to target removal and compression Artifacts. First, the network decomposes the compressed and distorted image into a texture layer (including HF component) and a structure layer (including LF component); secondly, it uses a multi-path recursive residual network to enhance texture and structure information respectively; finally, the structure and texture information are merged And feed it into the regression network to generate the final reconstructed image.
进一步地,在双流的同一支流内的每一个残差单元内采用权重共享的策略,每一个支流的训练参数量是固定的,等同于一个4层卷积神经网络的参数量。Further, the weight sharing strategy is adopted in each residual unit in the same tributary of the dual stream, and the training parameter amount of each tributary is fixed, which is equivalent to the parameter amount of a 4-layer convolutional neural network.
进一步地,首先通过最小化L
0范数的方法获得期望的结构层。其次计算出压缩失真图像与结构层的差异并作为相应的纹理层,其公式表示为:
Further, first obtain the desired structure layer by minimizing the L 0 norm. Secondly, the difference between the compressed and distorted image and the structure layer is calculated and used as the corresponding texture layer. The formula is expressed as:
I
lq=I
s+I
t
I lq =I s +I t
I
lq表示压缩失真图像,I
s表示对应于失真图像粗略信息的结构层,I
t表示对应于失真图像精细信息的纹理层。
I lq represents a compressed distorted image, I s represents a structural layer corresponding to the coarse information of the distorted image, and I t represents a texture layer corresponding to the fine information of the distorted image.
进一步地,所述网络架构由若干个递归残差单元(RRU,Recursive Residual Unit)和中间残差块(IRB,Intermediate Residual Block)组成,集成了全局残差学习、局部残差学习以及多路径中间残差学习三种残差学习;递归是指在各个特征映射之间使用相同权重,即参数共享。随着网络深度加深,网络学习的参数量也会线性增加,通过在每个递归单元之间权重共享来限制网络学习参数量。Further, the network architecture is composed of several recursive residual units (RRU, Recursive Residual Unit) and intermediate residual blocks (IRB, Intermediate Residual Block), which integrates global residual learning, local residual learning, and multi-path intermediate Residual learning There are three kinds of residual learning; recursion refers to the use of the same weight between each feature map, that is, parameter sharing. As the network deepens, the amount of network learning parameters will also increase linearly, and the amount of network learning parameters is limited by weight sharing between each recursive unit.
进一步地,所述RRU主要由两个卷积层和ReLU激活层组成,该RRU的表示公式为:Further, the RRU is mainly composed of two convolutional layers and a ReLU activation layer, and the expression formula of the RRU is:
X
u、X
u′表示第u个RRU的输入和输出,
表示RRU的映射函数,
是第u个RRU中第i个卷积层的映射函数,
表示第u个RRU中第i个卷积层的权重,函数σ是ReLU激活函数。
X u and X u ′ represent the input and output of the u-th RRU, Represents the mapping function of RRU, Is the mapping function of the i-th convolutional layer in the u-th RRU, Indicates the weight of the i-th convolutional layer in the u-th RRU, and the function σ is the ReLU activation function.
进一步地,所在每个IRB内,采用跳跃连接方式将低级特征传送给残差块的后一个网络层;假设每个IRB的表示函数为
第b个IRB的对应输入和输出分别为X
b和X
b′,
表示第b个递归块的第i个残差单元的权重;则由两个RRU组成IRB的公式表示为:
Further, in each IRB, the low-level features are transmitted to the next network layer of the residual block by skip connection mode; assuming that the representation function of each IRB is The corresponding input and output of the b-th IRB are X b and X b ′, Indicates the weight of the i-th residual unit of the b-th recursive block; the formula of IRB composed of two RRUs is expressed as:
因此,具有两个IRB的网络表示公式为:Therefore, the expression formula for a network with two IRBs is:
X和X′分别表示网络的输入与输出,f和f
rec分别表示整个双流网络内第一个和最后一个卷积层的映射函数。
表示所提出的网络的整体映射函数。
X and X'respectively represent the input and output of the network, f and f rec respectively represent the mapping function of the first and last convolutional layer in the entire dual-stream network. Represents the overall mapping function of the proposed network.
本发明的有益效果是:本发明在去除有损压缩图像伪影过程中,设计了一 种双流多路径递归残差网络的架构,网络中两个支流分别用于减少与高频或低频图像分量相关的特定类型伪影,并利用非线性回归网络将两个输出组合在一起,使得所提出的网络能够显着地抑制压缩伪影,减少失真,进一步增强重建的图像。The beneficial effect of the present invention is that in the process of removing artifacts from lossy compressed images, the present invention designs a dual-stream multi-path recursive residual network architecture. Related specific types of artifacts, and use a nonlinear regression network to combine the two outputs, so that the proposed network can significantly suppress compression artifacts, reduce distortion, and further enhance the reconstructed image.
附图说明Description of the drawings
下面结合附图和实施例对本发明进一步说明:The present invention will be further described below in conjunction with the drawings and embodiments:
图1是本发明的双流多路径递归残差网络的结构示意图;Figure 1 is a schematic structural diagram of the dual-stream multi-path recursive residual network of the present invention;
图2是本发明的递归残差单元的结构示意图。Fig. 2 is a schematic diagram of the structure of the recursive residual unit of the present invention.
具体实施方式detailed description
由于去块效应、去振铃伪影与锐化边缘(去模糊)任务之间相互抵消,如果不区分这些伪影时,端到端的卷积网络在减少一种类型伪影的同时会增强其他类型的伪影。本发明设计一种基于双流多路径递归残差网络架构,利用图像高频(HF,high frequency)和低频(LF,low frequency)分量来针对性的去除压缩伪影。首先,网络将压缩失真图像分解为纹理层(包含HF分量)和结构层(包含LF分量);其次,利用多路径递归残差网络分别增强纹理和结构信息;最终,将结构和纹理信息合并后并馈送到回归网络中以生成最终的重建图像。另外,为了克服网络训练困难,减少训练参数,在同一支流内的每一个残差单元内采用权重共享的策略。因此,每一个支流的训练参数量是固定的,等同于一个4层卷积神经网络的参数量。Since the tasks of deblocking, ringing artifacts, and edge sharpening (deblurring) cancel each other out, if these artifacts are not distinguished, the end-to-end convolutional network will reduce one type of artifact while enhancing the others. Type of artifacts. The present invention designs a dual-stream multi-path recursive residual network architecture, which uses high frequency (HF) and low frequency (LF) components of an image to specifically remove compression artifacts. First, the network decomposes the compressed and distorted image into a texture layer (including HF components) and a structure layer (including LF components); secondly, it uses a multi-path recursive residual network to enhance texture and structure information respectively; finally, the structure and texture information are combined And feed it into the regression network to generate the final reconstructed image. In addition, in order to overcome the difficulties of network training and reduce training parameters, a weight sharing strategy is adopted in each residual unit in the same tributary. Therefore, the amount of training parameters for each tributary is fixed, which is equivalent to the amount of parameters of a 4-layer convolutional neural network.
具体的方法或步骤如下:The specific methods or steps are as follows:
1、结构-纹理分解1. Structure-texture decomposition
首先,通过最小化L
0范数的方法获得期望的结构层。其次,计算出压缩失真图像与结构层的差异并作为相应的纹理层。
First, the desired structure layer is obtained by minimizing the L 0 norm. Secondly, the difference between the compressed and distorted image and the structure layer is calculated and used as the corresponding texture layer.
将压缩失真图像分解为结构层(包含LF分量)和纹理层(包含HF分量)。The compressed and distorted image is decomposed into a structure layer (including LF component) and a texture layer (including HF component).
其公式表示为:Its formula is expressed as:
I
lq=I
s+I
t
I lq =I s +I t
其中,I
lq表示压缩失真图像,I
s表示对应于失真图像粗略信息的结构层,I
t表示对应于失真图像精细信息的纹理层。
Among them, I lq represents a compressed distorted image, I s represents a structural layer corresponding to the coarse information of the distorted image, and I t represents a texture layer corresponding to the fine information of the distorted image.
2、结构流2. Structure flow
为了增强压缩图像结构层的细节信息,设计了基于多路径递归残差网络。整个网络由若干个递归残差单元(RRU,Recursive Residual Unit)和中间残差块(IRB,Intermediate Residual Block)组成,集成了全局残差学习、局部残差学习以及多路径中间残差学习三种残差学习。这类组合结构不仅可以帮助梯度和低级特征的传输,还可以显着减少网络的参数量。In order to enhance the detailed information of the compressed image structure layer, a recursive residual network based on multi-path is designed. The entire network is composed of several recursive residual units (RRU, Recursive Residual Unit) and intermediate residual blocks (IRB, Intermediate Residual Block), which integrates global residual learning, local residual learning and multi-path intermediate residual learning. Residual learning. This kind of combined structure can not only help the transmission of gradients and low-level features, but also significantly reduce the amount of network parameters.
1)递归残差单元1) Recursive residual unit
递归是指在各个特征映射之间使用相同权重,即参数共享。随着网络深度加深,网络学习的参数量也会线性增加,因此,可以通过在每个递归单元之间权重共享来限制网络学习参数量。RRU主要由两个卷积层和ReLU激活层组成,其结构示意图如图2所示。与ResNet内残差单元具有相同的结构,区别在于激活顺序的不同。在ResNet的每个残差单元中,激活函数在卷积层之后执行,而RRU则在卷积层之前执行激活层(BN和ReLU)。因此,该RRU的表示公式为:Recursion refers to the use of the same weight between each feature map, that is, parameter sharing. As the network deepens, the amount of network learning parameters will also increase linearly. Therefore, the amount of network learning parameters can be limited by weight sharing between each recursive unit. RRU is mainly composed of two convolutional layers and ReLU activation layer, and its structure diagram is shown in Figure 2. It has the same structure as the residual unit in ResNet, but the difference lies in the activation sequence. In each residual unit of ResNet, the activation function is executed after the convolutional layer, and the RRU executes the activation layer (BN and ReLU) before the convolutional layer. Therefore, the expression formula of this RRU is:
其中,X
u、X
u′表示第u个RRU的输入和输出,
表示RRU的映射函数,
是第u个RRU中第i个卷积层的映射函数,
表示第u个RRU中第i个卷积层的权重,函数σ是ReLU激活函数。
Among them, X u and X u′ represent the input and output of the u-th RRU, Represents the mapping function of RRU, Is the mapping function of the i-th convolutional layer in the u-th RRU, Indicates the weight of the i-th convolutional layer in the u-th RRU, and the function σ is the ReLU activation function.
2)区间残差块2) Interval residual block
在每个IRB内,采用跳跃连接方式将低级特征传送给残差块的后一个网络层。因此,在全局连接和局部连接之间存在多条跳跃连接,如图1中中间下方弯曲箭头所示。假设每个IRB的表示函数为
第b个IRB的对应输入和输出分别为X
b和X
b′,
表示第b个递归块的第i个残差单元的权重。则由两个RRU 组成IRB的公式表示为:
In each IRB, the low-level features are transmitted to the next network layer of the residual block by skip connection mode. Therefore, there are multiple jump connections between the global connection and the local connection, as shown by the curved arrow in the lower middle in Figure 1. Assume that the representation function of each IRB is The corresponding input and output of the b-th IRB are X b and X b ′, Indicates the weight of the i-th residual unit of the b-th recursive block. Then the formula of IRB composed of two RRUs is expressed as:
因此,具有两个IRB的网络表示公式为:Therefore, the expression formula for a network with two IRBs is:
其中,X和X′分别表示网络的输入与输出,f和f
rec分别表示整个双流网络内第一个和最后一个卷积层的映射函数。
表示所提出的网络的整体映射函数。3)网络结构
Among them, X and X'respectively represent the input and output of the network, and f and f rec respectively represent the mapping function of the first and last convolutional layer in the entire dual-stream network. Represents the overall mapping function of the proposed network. 3) Network structure
由于三种残差学习,以及RRU和IRB的设计,本发明提出的网络具有灵活的组合结构。给定一个特定的网络层数,可以自由地调整RRU的数量和IRB的数量。假设RRU的数量表示为U,IRB的数量表示为B,则网络层数计算公式如下:Due to the three kinds of residual learning and the design of RRU and IRB, the network proposed by the present invention has a flexible combined structure. Given a specific number of network layers, the number of RRUs and the number of IRBs can be adjusted freely. Assuming that the number of RRUs is denoted as U and the number of IRBs is denoted as B, the calculation formula for the number of network layers is as follows:
d=2+2×U×Bd=2+2×U×B
假设设计的网络层数d=20,那么该网络结构具有以下三个不同的类型:Assuming that the number of designed network layers d=20, then the network structure has the following three different types:
A、1B9U:仅仅只有一个IRB块,且该IRB内含有9个RRU;A. 1B9U: There is only one IRB block, and the IRB contains 9 RRUs;
B、3B3U:总共有3个IRB块,且每个IRB内包含3个RRU;B, 3B3U: There are a total of 3 IRB blocks, and each IRB contains 3 RRUs;
C、9B1U:总共有9个IRB块,且每个IRB中只含有一个RRU。C. 9B1U: There are a total of 9 IRB blocks, and each IRB contains only one RRU.
由于3B3U的组合结构包含三种残差学习,因此,3B3U组合结构分别应用于本发明设计的结构流和纹理流。特别地是,随着网络层数增加,组合结构就越灵活,即存在更多不同的组合。Since the combined structure of 3B3U includes three kinds of residual learning, the combined structure of 3B3U is respectively applied to the structure flow and texture flow designed in the present invention. In particular, as the number of network layers increases, the combination structure becomes more flexible, that is, there are more different combinations.
3、纹理流和结构流回归网络3. Texture flow and structure flow regression network
结构层的目的在于恢复图像丢失的高频信息。相反地,纹理层的处理过程旨在去除压缩伪影,并保留原始图像的边缘等细节信息。对原始真实图像的纹理层进行监督学习,通过所设计的包含递归残差单元区和间残差块的网络结构能够极大的抑制纹理层中的强块效应和振铃伪影。The purpose of the structure layer is to restore the high-frequency information lost in the image. On the contrary, the processing process of the texture layer aims to remove compression artifacts and preserve the details such as the edges of the original image. Supervised learning is performed on the texture layer of the original real image, and the designed network structure including the recursive residual unit area and the inter-residual block can greatly suppress the strong blocking effect and ringing artifacts in the texture layer.
本发明设计的网络结构里,结构流和纹理流两个支流并行运作,分别输出相应 信息增强后结构层
和纹理层
然后,两个支流相应的输出进行逐个像素相加以获得增强的图像,即
最后,增强的图像
被送入非线性回归网络以进一步改善重建的图像,该回归网络的结构与应用于结构或纹理流的网络相同。
In the network structure designed by the present invention, the two branches of the structure flow and the texture flow are operated in parallel, and the corresponding information is outputted and the structure layer is enhanced. And texture layer Then, the corresponding outputs of the two tributaries are added pixel by pixel to obtain an enhanced image, namely Finally, the enhanced image It is fed into a non-linear regression network to further improve the reconstructed image. The structure of the regression network is the same as the network applied to the structure or texture stream.