CN112365554A - Compressed sensing image reconstruction method based on multi-scale residual error neural network - Google Patents
Compressed sensing image reconstruction method based on multi-scale residual error neural network Download PDFInfo
- Publication number
- CN112365554A CN112365554A CN202011153555.1A CN202011153555A CN112365554A CN 112365554 A CN112365554 A CN 112365554A CN 202011153555 A CN202011153555 A CN 202011153555A CN 112365554 A CN112365554 A CN 112365554A
- Authority
- CN
- China
- Prior art keywords
- layer
- reconstruction
- network
- output
- msrb
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000013528 artificial neural network Methods 0.000 title claims description 10
- 239000013598 vector Substances 0.000 claims abstract description 40
- 238000005070 sampling Methods 0.000 claims abstract description 34
- 101001116774 Homo sapiens Methionine-R-sulfoxide reductase B2, mitochondrial Proteins 0.000 claims abstract description 14
- 102100024862 Methionine-R-sulfoxide reductase B2, mitochondrial Human genes 0.000 claims abstract description 14
- 238000005259 measurement Methods 0.000 claims description 25
- 239000011159 matrix material Substances 0.000 claims description 22
- 238000012549 training Methods 0.000 claims description 20
- 238000004422 calculation algorithm Methods 0.000 claims description 10
- 238000005457 optimization Methods 0.000 claims description 10
- 230000006870 function Effects 0.000 claims description 9
- 238000007493 shaping process Methods 0.000 claims description 9
- 210000002569 neuron Anatomy 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 7
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 3
- 230000001131 transforming effect Effects 0.000 claims description 3
- 230000001143 conditioned effect Effects 0.000 claims description 2
- 230000006835 compression Effects 0.000 abstract description 9
- 238000007906 compression Methods 0.000 abstract description 9
- 238000012545 processing Methods 0.000 abstract description 6
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 abstract description 5
- 230000000694 effects Effects 0.000 abstract description 3
- 238000013527 convolutional neural network Methods 0.000 description 16
- 238000012360 testing method Methods 0.000 description 6
- 238000013135 deep learning Methods 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 241000287530 Psittaciformes Species 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/061—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using biological neurons, e.g. biological neurons connected to an integrated circuit
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- General Physics & Mathematics (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Neurology (AREA)
- Microelectronics & Electronic Packaging (AREA)
- Image Analysis (AREA)
Abstract
The invention belongs to the field of image processing, and aims to improve the reconstruction quality and the reconstruction effect of an image to the maximum extent. The compression sampling model comprises a remolded reshape layer/a full connection layer; an initial reconstruction model, wherein one full-connection layer FC is used for up-sampling an observation vector y to finally obtain a model xvAnother reshaped reshape layer rearranges the output of the upsampling process to form a preliminary reconstructed image xir; further processing xir using a deep multi-scale residual reconstruction model comprising a plurality of multi-scale residual blocks MSRBs, the output features of each MSRB being mapped to the networkAnd the terminal links the characteristic information and the output of the initial reconstruction model into the network terminal. The invention is mainly applied to the image processing occasion.
Description
Technical Field
The invention belongs to the field of image processing, and relates to the structure optimization of a convolutional neural network in deep learning, image compression sampling and reverse reconstruction research. In particular to a compressed sensing image reconstruction method based on a multi-scale residual error neural network.
Background
Compressed Sensing (CS) theory is a very promising emerging technology that demonstrates that the original signal [1,2] can be reconstructed with a high probability with less sampled data than required by nyqiust sampling theory when the signal is sparse in the transform domain. In addition, the CS theory can complete the sampling and compression processing of the signals at the same time, and is beneficial to relieving the pressure of hardware in data acquisition, storage and transmission. Based on the advantages, the CS theory is widely applied to a plurality of practical applications such as medical image scanners [3], single-pixel cameras [4], cognitive radio communication [5] and the like.
In order to fully exploit the potential of the CS theory, many image reconstruction methods based on the CS theory have been proposed in the last decade. The reconstruction method of the compressed sensing image can be roughly divided into two types: the conventional compressed sensing image reconstruction method based on optimization and the latest compressed sensing image reconstruction method based on CNN (convolutional neural network). To better analyze these methods and to show the advantages of our method, we first explain the compressed sensing of the signal and its inverse reconstruction process in mathematical form: the purpose of compressed sensing and inverse reconstruction theory is to obtain the corresponding compressed observed quantity y ═ x ∈ R from the original signalMIn-process reverse reconstruction of original signal x epsilon RNWhere φ ∈ RM×NCalled measurement matrix, for implementing a compressive sampling process on a raw signal x, y ∈ RMIs an observation vector obtained from compressed samples in the original signal x according to the compressed sensing theory. Due to m<<n, reconstructing x from y back is a highly ill-posed problem.
For this problem, most conventional compressed sensing image reconstruction methods [6,7,8,9 ]]Assuming that the original image signal is structurally sparse in some transform domains, iterative calculations are then applied to solve an optimization problem. However, natural images in the real world do not have exact sparsity in their transform domain, which limits the reconstruction performance of these algorithms. In addition, the iterative computation has high computational complexity and long time for executing the reconstruction task, which also limits the real-time application of the image compression perception theory. Recently, CN is receivingN application to image super-resolution reconstruction [10 ]]Inspiring in the field, researchers developed a new class of compressed sensing image reconstruction methods based on CNN. Kulkarni et al first proposed a method using a simple convolutional neural network ReconvNet [11]And recovering the original image from the observation vector in the reverse direction. Yao et al [12]]Further perfecting the structure of the reconstructed network and forming a new network DR2-Net[12]The full connection layer is introduced as a linear mapping model, and the residual error network is used as a reconstruction model. In document [13]]Zhang et al propose a CNN model called ISTA-Net, so as to appropriately fuse the structural advantages of the conventional iterative algorithm and the reconstruction speed advantages of the CNN method.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to realize the high-efficiency and high-quality reconstruction and restoration of the compressed sensing image through a multi-scale residual error neural network, and the reconstruction quality and the reconstruction effect of the image are improved to the maximum extent on the premise of controlling the reconstruction time cost of the compressed sensing image. Therefore, the technical scheme adopted by the invention is that the compressed sensing image reconstruction system based on the multi-scale residual error neural network comprises the following steps: the compressively sampled model contains a remoulded reshape layer for size transforming an input image block xp of size n × n, resulting in a single column vector x of size n2 × 1v(ii) a Then one full connection layer is used for pair vector xvPerforming a compressive sampling process and generating a corresponding size of α × n2The image observation vector y, a represents the measurement rate, i.e. the compressed sampling rate, and then the output of the compressed sampling model is fed into the initial reconstruction model, wherein a full-connected layer FC is used for up-sampling the observation vector y, the FC (full-connected layer) contains n2Each neuron finally obtaining the sum of xvAnother reshaped reshape layer rearranges the output of the upsampling process to form a preliminary reconstructed image xir; further processing xir with a deep multi-scale residual reconstruction model comprising a plurality of multi-scale residual blocks MSRBs, mapping the output characteristics of each MSRB to the end of the network, and connecting the characteristic information with the output of the initial reconstruction model into the networkThe end of the collaterals.
Wherein, compressing the sampling model:
the input image block xp is n x n in size, and the shaping operation of the shaping layer in the compressed sample model is fre1The reshape layer is used to reshape the size of the input image block while preserving the value of each pixel, and the output of the reshape layer is then expressed as:
xv＝fre1(xp) (1)
wherein xvIs provided with n2A single column vector of size x 1. Second, at compressed sample xvAdding a full connection layer comprising alpha x n2Nerve, α represents the measurement rate; the output of the full link layer is y, then y and xvThe relationship between them is expressed as:
y＝ffull1(xv) (2)
wherein f isfull1An operation representing full connectivity;
this full connectivity layer performs a similar function to the conventional random gaussian measurement matrix: in a layer of2Each neuron generating an AND vector xvThe weight vectors with the same size are trained on the network to adaptively learn the value of the weight vector, and then the vector x is used for outputting the current neuronvAnd inner product representation of weight vector, thereby a layer of alphaxn2The output of each neuron may be represented by a vector xvAnd one is composed of. alpha.x n2An inner product of weight matrix formed by the weight vectors, wherein the weight matrix is (alpha x n2) x n2Size;
the output of the compressed sampling model is sent to the next initial reconstruction model, whose output is denoted xirThen y and xirThe relationship between can be expressed as:
xir＝fre2(ffull2(y)) (3)
wherein f isfull2Representing operations in a fully connected layer, can be seen as simulating from y to the original signal xvLinear mapping of the initial reconstruction process, and fre2Indicating a shaping operation in the reshape layer.
Wherein the multi-scale residual error reconstruction module MSRB:
there are three convolution channels in the MSRB, each channel extracting some scale information from the input feature map using a fixed size convolution kernel, and further sharing and merging the image information extracted from the three channels.
The above operation formula of the MSRB unit is expressed as follows:
where F denotes the output of a certain convolutional Layer (Conv Layer) in the MSRB unit, w and b denote the weight and offset in the convolutional Layer, respectively, the superscript denotes the position of the convolutional Layer in the MSRB unit, and the subscript denotes the size of the convolutional core in the convolutional Layer. Mi-1And MiRepresenting the input and output of the ith MSRB unit. []To representThe channel is connected to a conditioned operation, max (0, x) denotes Relu [16 ]]A function. w is aoAnd boRespectively representing the weight and the deviation of the last convolution layer of the MSRB unit;
assuming that the number of MSRB structural units in the MSRNet is n, the weight sum deviation of the last convolution layer is wfAnd bfThen the final reconstruction result x of the MSRNetfrExpressed as:
xfr＝max(0,wf*[M1,……,MN]+bf)+xir (11)。
training and execution of MSRNet network:
for the training dataset, we use the same document [10,11 ]]The same dataset, comprising 91 images, these original images are rotated 90, 180, 270 to expand and enhance the dataset, 33 x 33 image patches are extracted from the original images by tiling, the tiling step size is set to 14, a network training set is made, for the training strategy, the mean square error MSE is used as the loss function to minimize the predicted loss of the network, and Adam is used as the network optimization algorithm to train the network, the initial learning rate is set to 0.001, the learning rate is reduced by a factor of 10 every 5 ten thousand, the learning rate is less than 10-5The network training process is terminated.
The invention has the characteristics and beneficial effects that:
the end-to-end multi-scale residual error reconstruction network provided by the method realizes the reverse reconstruction process of the compressed sensing image. Through training of the convolutional neural network based on end-to-end optimization, the difficulty of generating a compression measurement matrix is greatly reduced, and meanwhile, the reconstruction precision of the network is improved. In addition, multi-scale residual error learning is introduced, the learning capability of multi-scale information and characteristics of the network is improved, better image reconstruction quality is facilitated, and meanwhile, lower image reconstruction time cost is kept. Experiments on a standard image test set show that the MSRNET is an efficient and excellent compressed sensing image reconstruction system.
Description of the drawings:
FIG. 1 shows that PSNR values of image parrots are reconstructed by different algorithms under the condition that the compression measurement rate is 10%And time costs. Reconstruction results of our method compare RecoNet [11]And DR2-net[12]The time cost is well controlled because the time cost is respectively 5.61dB and 4.30dB higher.
FIG. 2 image compressive sensing and inverse reconstruction process based on multi-scale residual error neural network
FIG. 3: multi-scale residual Unit (MSRB) Structure representation
FIG. 4: the previous frame of image is the reconstruction result under the condition that the measurement rate is 25%, and the next frame of image is the reconstruction result under the condition that the measurement rate is 4%.
Detailed Description
For existing CNN-based models, they are typically able to complete the reconstruction task very quickly due to the strong learning capabilities of CNNs. However, these existing CNN-based methods often use simple and basic network frameworks such as residual networks or deep networks to form a reconstruction model, and cannot fully utilize and mine the learning and mapping capabilities of complex convolutional networks. In order to improve the learning capability of the reconstruction network and further improve the recovery performance of the compressed sensing image, a new multi-scale residual error reconstruction network MSRNet is proposed. In MSRNet, we construct a multi-scale residual block (MSRB) as the basic block for reconstructing the network. There are three parallel channels of convolution kernels of different sizes in one MSRB, each of which can be used to extract one scale information from the input feature map. In addition, the information extracted by the current channel is shared with other channels, so that the network learning capability of the characteristic information with different scales is obviously improved. In addition, the MSRNet also introduces jump connection and residual learning, and the prediction accuracy of the network is improved. Another contribution of MSRNet is the measurement matrix. In previous CNN-based reconstruction methods, the corresponding compressed perceptual observation vector y was generated by compression sampling the original image x with a random measurement matrix of appropriate size, which was artificially generated before training the network. However, in MSRNet, we use a compressive sampling network to implement the compressive sampling process, and this model can perform parameter update and learning by training the network, which has two advantages: (1) generating a matrix of random nature is very difficult for hardware. By training the MSRNet, a deterministic measurement matrix can be generated, which is self-learned by the network and is easy to implement in hardware. (2) By integrating the compressive sampling and reverse recovery processes into one network, we actually build and optimize a complete end-to-end CNN, rather than optimizing the image compressive sensing and reconstruction components separately.
Overall, our MSRNet's major contributions are three areas: (1) multi-scale learning: through the multi-scale convolution channel, the network can extract and utilize feature information of different scales, and has stronger learning ability. (2) End-to-end optimization: a complete end-to-end network is realized to simulate image compression and reverse reconstruction, and the difficulties of network optimization and hardware realization are avoided. (3) Accuracy and time complexity: as shown in fig. 1, our MSRNet achieves significant improvement in reconstruction performance with lower temporal complexity over the standard image set.
As an end-to-end network that simulates the process of image compressed sensing and reverse reconstruction, we designed MSRNet to include three sub-models: the device comprises a compression sampling model, an initial reconstruction model and a deep multi-scale residual error reconstruction model.
The method comprises the following specific steps:
as shown in fig. 2, the three submodels are cascaded and integrated to form a complete MSRNet. In MSRNet, the compressed sampling model comprises a reshaping (reshape) layer for transforming the size of an input image block xp of size n × n, resulting in a single-column vector x of size n2 × 1vThen one full connection layer is used for pair vector xvPerforming a compressive sampling process and generating a corresponding size of α × n2And (α represents the measurement rate, i.e., the compressed sampling rate) of the image observation vector y. The output of the compressed sample model is then fed into an initial reconstruction model, where a fully-connected layer (FC) contains n2Neuron) for up-sampling the observation vector y to obtain the sum xvWith the same size of the output data, another reshaping (reshape) layer rearranges the output of the upsampling process to form a preliminary reconstructed image xir. Considering the initialThe reconstructed image xir is not ideal in terms of PSNR values, human vision and the like, and the performance and visual quality of the reconstructed image are further improved by using a deep multi-scale residual reconstruction model including a plurality of multi-scale residual blocks (MSRB). In addition, the output characteristic mapping of each MSRB is sent to the end of the network by using skip connection, and the characteristic information is connected with the output of the initial reconstruction model and sent to the end of the network, so that the reconstruction performance of the network is improved.
We now analyze and present the compressive sampling and deep multi-scale residual reconstruction modules in detail.
Compressive sampling model in MSRNet
In previous methods, researchers have implemented sampling operations using a random gaussian matrix as the measurement matrix phi. Although this is an effective compressive sampling measurement matrix, it is difficult to implement a random matrix in practical applications. In MSRNet, we propose a compressed sampling model to replace the random gaussian matrix.
Assuming that the input image block xp is n × n in size, the shaping operation of the shaping layer in the compressed sample model is fre1The reshape layer is used to reshape the size of the input image block while preserving the value of each pixel, the output of the reshape layer can be expressed as:
xv＝fre1(xp) (1)
wherein xvIs provided with n2A single column vector of size x 1. Second, at compressed sample xvAdding a full connection layer comprising alpha x n2Nerves (α represents the measurement rate). Assuming that the output of the fully connected layer is y, then y and xvThe relationship between can be expressed as:
y＝ffull1(xv) (2)
wherein f isfull1Indicating a fully connected operation.
This full connectivity layer performs a similar function to the conventional random gaussian measurement matrix: in a layer of2Each neuron generating an AND vector xvWeight vectors of the same size. Through a network pairAdaptively learning the value of the weight vector, and then using the vector x as the output of the current neuronvAnd inner product representation of weight vector, thereby a layer of alphaxn2The output of each neuron may be represented by a vector xvAnd one is composed of. alpha.x n2The inner product of the weight matrix formed by the individual weight vectors is represented. Wherein the weight matrix is (alpha x n2) x n2The size is the same as the size of the previous conventional measurement matrix. However, in MSRNET, the weight matrix is generated by training a network based on end-to-end optimization, which is more advantageous in terms of hardware implementation and compressive sampling efficiency.
The output of the compressed sampling model is sent to the next initial reconstruction model, the output of which can be represented as xirThen y and xirThe relationship between can be expressed as:
xir＝fre2(ffull2(y)) (3)
wherein f isfull2Representing operations in a fully connected layer, can be seen as simulating from y to the original signal xvLinear mapping of the initial reconstruction process, and fre2Indicating a shaping operation in the reshape layer.
Second, multi-scale residual error reconstruction module in MSRNet
In MSRNet, we constructed a multi-scale residual block (MSRB) as the basic building block of the deep reconstruction model. As shown in fig. 2, there are three convolution channels in the MSRB, each channel extracting some scale information from the input feature map using a fixed size convolution kernel. In addition, the image information extracted from the three channels can be further shared and fused, and the learning and expression capacity of the network on different scale features is improved.
The above operation of the MSRB unit in fig. 2 can be formulated as follows:
where F denotes the output of a certain convolutional Layer (Conv Layer) in the MSRB unit, w and b denote the weight and offset in the convolutional Layer, respectively, the superscript denotes the position of the convolutional Layer in the MSRB unit, and the subscript denotes the size of the convolutional core in the convolutional Layer. Mi-1And MiRepresenting the input and output of the ith MSRB unit. []Denotes a channel connect (connected) operation, max (0, x) denotes Relu [16 ]]A function. w is aoAnd boRespectively representing the weight and offset of the last convolutional layer of the MSRB unit.
Assuming that the number of MSRB structural units in the MSRNet is n, the weight sum deviation of the last convolution layer is wfAnd bfThen the final reconstruction result of MSRNet (denoted as x)fr) Can be expressed as:
xfr＝max(0,wf*[M1,……,MN]+bf)+xir (11)
training and execution details for a three, MSRNet network
For the training dataset, we use the same document [10,11 ]]The same data set comprising 91 images. Receiver [10 ]]Inspired, we rotate these original images 90, 180, 270 to expand and enhance the data set. According to the common practice of the existing CNN-based compressed sensing method, 33 x 33 image small blocks are extracted from an original image by block cutting, the block cutting step length is set to be 14, and a network training set is manufactured. For the training strategy, we use Mean Square Error (MSE) as a loss function to minimize the predicted loss of the network, and Adam [17 ]]Our network is trained as a network optimization algorithm. The initial learning rate is set to 0.001, the learning rate is reduced by 10 times every 5 ten thousand times, and the learning rate is less than 10-5The network training process is terminated.
In addition, the number of convolution kernels for each convolution layer in MSRNet is set to 64, except for the last convolution layer, which is set to 1, to form the final reconstructed image. The batch size (batch size) of the image patch fed into the network training is set to 64. All experiments were done on a 12GB memory Titanx GPU through the deep learning framework, caffe [18 ].
Like other compressed sensing methods, we used Set11[11] as the test dataset and tested the reconstruction performance of the different methods at four measurement rates (MR ═ 0.25,0.1,0.04, 0.01). Considering the balance between performance and computational complexity, we performed our experiments using msrnets containing 2 MSRBs. We compared our approach to the most advanced existing approaches, including TVAL3[7], D-AMP [14], SDA [15], Reconnet [11], DR2-NET [12], ISTA-NET [13] the average PSNR results on the Set11[11] test Set are reported in Table 1:
table 1: PSNR values of reconstruction results of different methods at different measurement rates (Set11 test Set)
As can be seen from table 1, our MSRNet significantly improved the reconstruction performance over the previous best methods, e.g., MR 0.25, and our method was 7.82db and 4.7db higher than CNN-based ReconNet [11] and DR2-Net [12], respectively. For MR ═ 0.01, our method improves PSNR by 2.78db over ISTA-Net [13], and MSRNet is also robust to reconstruction of compressed perceptual images, even at very small compressed measurement rates, MSRNet can achieve the currently optimal reverse reconstruction effect of compressed perceptual images, more reconstructed example images at different measurement rates are shown in fig. 4. Our method produces detailed texture and edges in the reconstructed image with better visual effect than other known methods.
For the temporal complexity of the algorithm, we used a single frame image of 256 × 256 size as the test image to calculate the reconstruction time for the different methods, the results are shown in table 2.
Table 2: different methods realize the time required by the reconstruction of a single frame (256X256) image under different measurement rates
For fair comparison, we performed all tests on similar performing devices, and from Table 2 we see that our method compares to the conventional image compressive sensing reconstruction algorithm TVAL3[7]]，D-AMP[14]Much faster. SDA [15]]As a non-iterative algorithm based on deep learning, the method has a great speed advantage compared with other compressed sensing image reconstruction algorithms based on CNN (compressed sensing network), although the method is more reconNet than Reconnet [11]]Slow by 5-6 times, ratio DR2-NET[12]The speed is 2 times slower, but the time complexity of the MSRNet is acceptable and can meet the requirements of practical application, so that the limited increase of the time complexity is acceptable in consideration of the great advantage of reconstruction performance of the method.
Reference to the literature
[1]E.J.Candes,T.Tao,“Near-optimal signal recovery from random projections:Universal encoding strategies？,”IEEE Transactions on Information Theory,52(12):5406-5425,2006.
[2]M.F.Duarte,M.A.Davenport,D.Takbar,J.N.Laska,T.Sun,K.F.Kelly,and R.G.Baraniuk,“Single-pixel imaging via compressive sampling,”IEEE Signal Processing Magazine,25(2):83–91,2008.
[3]M.Lustig,D.Donoho,and J.M.Pauly,“Sparse mri:The application of compressed sensing for rapid mr imaging,”Magnetic Resonance in Medicine,58(6):1182–1195,2007.
[4]F.Rousset,N.Ducros,A.Farina,G.Valentini,C.DAndrea,F.Peyrin,“Adaptive basis scan by wavelet prediction for single-pixel imaging,”IEEE Transactions on Computational Imaging,3(1):36–46,2017.
[5]S.K.Sharma,E.Lagunas,S.Chatzinotas,and B.Ottersten,“Application of compressive sensing in cognitive radio communications:A survey,”IEEE Communication Surveys&Tutorials,18(3):1838–1860,2016.
[6]Y.Kim,M.S.Nadar,and A.Bilgin,“Compressed sensing using a gaussian scale mixtures model in wavelet domain,”In ICIP,IEEE,pages 3365–3368.,2010.
[7]C.Li,W.Yin,H.Jiang,and Y.Zhang,“An efficient augmented lagrangian method with applications to total variation minimization,”Computational Optimization and Applications,56(3):507–530,2013.
[8]C.A.Metzler,A.Maleki,and R.G.Baraniuk,“From denoisingto compressed sensing,”IEEE Transactions on Information Theory,62(9):5117–5144,2016.
[9]J.Zhang,D.Zhao,and W.Gao,“Group-based sparse representation for image restoration,”IEEE Transactions on Image Processing,23(8):3336–3351,2014.
[10]C.Dong,C.C.Loy,K.He,and X.Tang,“Learning a deep convolutional network for image superresolution,”In Euro.Conf.Comp.Vision,Springer,2014,pages.184-199.
[11]K.Kulkarni,S.Lohit,P.Turaga,R.Kerviche,and A.Ashok,“Reconnet:Non-iterative reconstruction of images from compressively sensed random measurements,”CVPR,2016.
[12]H.Yao,F.Dai,D.Zhang,Y.Ma and S.Zhang,“DR2-Net:deep residual reconstruction network for image compressive sensing,”IEEE Conference on Computer Vision and Pattern Recognition,2017.
[13]J.Zhang,B.Ghanem,“ISTA-Net:Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing,”CVPR,2018.
[14]C.A.Metzler,A.Maleki,and R.G.Baraniuk,“From denois-ing to compressed sensing,”IEEE Trans.Information Theory,2016,62(9):5117–5144.
[15]A.Mousavi,A.B.Patel,and R.G.Baraniuk,“A deep learning approach to structured signal recovery,”in 2015 53rd Annual Allerton Conference on Communication,Control,and Computing,IEEE,2015,pp.1336–1343.
[16]X.Glorot,A.Bordes,and Y.Bengio,“Deep sparse rectifier neural networks,”In AISTATS,2011.3.
[17]D.Kingma,J.Ba.Adam,“A method for stochastic optimization,”ICLR,2014.5.
[18]Y.Jia,E.Shelhamer,J.Donahue,S.Karayev,J.Long,R.Girshick,S.Guadarrama,and T.Darrell,“Caffe:Convolutional architecture for fast feature embedding,”arXiv:1408.5093,2014.5。
Claims (5)
1. A compressed sensing image reconstruction system based on a multi-scale residual error neural network is characterized by comprising: the compressively sampled model contains a remoulded reshape layer for size transforming an input image block xp of size n × n, resulting in a single column vector x of size n2 × 1v(ii) a Then one full connection layer is used for pair vector xvPerforming a compressive sampling process and generating a corresponding size of α × n2The image observation vector y, a represents the measurement rate, i.e. the compressed sampling rate, and then the output of the compressed sampling model is fed into the initial reconstruction model, wherein a full-connected layer FC is used for up-sampling the observation vector y, the FC (full-connected layer) contains n2Each neuron finally obtaining the sum of xvAnother reshaped reshape layer rearranges the output of the upsampling process to form a preliminary reconstructed image xir; using blocks containing multiple multi-scale residuesThe deep multi-scale residual reconstruction model of the MSRB is further processed xir and the output features of each MSRB are mapped to the end of the network and these features are concatenated with the output of the initial reconstruction model to the end of the network.
2. The multi-scale residual neural network-based compressed sensing image reconstruction system of claim 1, wherein the compressed sampling model:
the input image block xp is n x n in size, and the shaping operation of the shaping layer in the compressed sample model is fre1The reshape layer is used to reshape the size of the input image block while preserving the value of each pixel, and the output of the reshape layer is then expressed as:
xv＝fre1(xp) (1)
wherein xvIs provided with n2A single column vector of size x 1. Second, at compressed sample xvAdding a full connection layer comprising alpha x n2Nerve, α represents the measurement rate; the output of the full link layer is y, then y and xvThe relationship between them is expressed as:
y＝ffull1(xv) (2)
wherein f isfull1An operation representing full connectivity;
this full connectivity layer performs a similar function to the conventional random gaussian measurement matrix: in a layer of2Each neuron generating an AND vector xvThe weight vectors with the same size are trained on the network to adaptively learn the value of the weight vector, and then the vector x is used for outputting the current neuronvAnd inner product representation of weight vector, thereby a layer of alphaxn2The output of each neuron may be represented by a vector xvAnd one is composed of. alpha.x n2An inner product of weight matrix formed by the weight vectors, wherein the weight matrix is (alpha x n2) x n2Size;
the output of the compressed sampling model is sent to the next initial reconstruction model, whose output is denoted xirThen y and xirThe relationship between can be expressed as:
xir＝fre2(ffull2(y)) (3)
wherein f isfull2Representing operations in a fully connected layer, can be seen as simulating from y to the original signal xvLinear mapping of the initial reconstruction process, and fre2Indicating a shaping operation in the reshape layer.
3. The multi-scale residual neural network-based compressed sensing image reconstruction system of claim 1, wherein the multi-scale residual reconstruction module MSRB: there are three convolution channels in the MSRB, each channel extracting some scale information from the input feature map using a fixed size convolution kernel, and further sharing and merging the image information extracted from the three channels.
4. The system of claim 4, wherein the MSRB unit is expressed by the following formula:
where F denotes the output of a certain convolutional Layer (Conv Layer) in the MSRB unit, w and b denote the weight and offset in the convolutional Layer, respectively, the superscript denotes the position of the convolutional Layer in the MSRB unit, and the subscript denotes the size of the convolutional core in the convolutional Layer. Mi-1And MiRepresenting the input and output of the ith MSRB unit. []Indicates the channel connection conditioned operation, max (0, x) indicates Relu [16 ]]A function. w is aoAnd boRespectively representing the weight and the deviation of the last convolution layer of the MSRB unit;
assuming that the number of MSRB structural units in the MSRNet is n, the weight sum deviation of the last convolution layer is wfAnd bfThen the final reconstruction result x of the MSRNetfrExpressed as:
xfr＝max(0,wf*[M1,……,MN]+bf)+xir (11)。
5. the multi-scale residual neural network-based compressed sensing image reconstruction system of claim 1, wherein the MSRNet network is trained and executed to: for a training dataset, the usage dataset comprises 91 images, these original images are rotated 90, 180, 270 to expand and enhance the dataset, 33 x 33 image patches are extracted from the original images by dicing, the dicing step size is set to 14, a network training set is made, for the training strategy, the mean square error MSE is used as a loss function to minimize the prediction loss of the network, and Adam is used as a network optimization algorithm to train the network, the initial learning rate is set to 0.001, the learning rate is reduced by 10 times every 5 ten thousand times, the learning rate is less than 10-5The network training process is terminated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011153555.1A CN112365554A (en) | 2020-10-26 | 2020-10-26 | Compressed sensing image reconstruction method based on multi-scale residual error neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011153555.1A CN112365554A (en) | 2020-10-26 | 2020-10-26 | Compressed sensing image reconstruction method based on multi-scale residual error neural network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112365554A true CN112365554A (en) | 2021-02-12 |
Family
ID=74512162
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011153555.1A Pending CN112365554A (en) | 2020-10-26 | 2020-10-26 | Compressed sensing image reconstruction method based on multi-scale residual error neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112365554A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113034414A (en) * | 2021-03-22 | 2021-06-25 | 上海交通大学 | Image reconstruction method, system, device and storage medium |
| CN113077044A (en) * | 2021-03-18 | 2021-07-06 | 北京工业大学 | General lossless compression and acceleration method for convolutional neural network |
| CN113658282A (en) * | 2021-06-25 | 2021-11-16 | 陕西尚品信息科技有限公司 | Image compression and decompression method and device |
| CN113688747A (en) * | 2021-08-27 | 2021-11-23 | 国网浙江省电力有限公司双创中心 | Method, system, device and storage medium for detecting personnel target in image |
| CN114066825A (en) * | 2021-10-29 | 2022-02-18 | 浙江工商大学 | Improved complex texture image flaw detection method based on deep learning |
| CN114494569A (en) * | 2022-01-27 | 2022-05-13 | 光线云(杭州)科技有限公司 | Cloud rendering method and device based on lightweight neural network and residual streaming transmission |
| CN115330635A (en) * | 2022-08-25 | 2022-11-11 | 苏州大学 | Image compression artifact removing method and device and storage medium |
| CN116468812A (en) * | 2023-05-16 | 2023-07-21 | 山东省计算中心（国家超级计算济南中心） | Image compressed sensing reconstruction method and system based on multiple branches and multiple scales |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107358576A (en) * | 2017-06-24 | 2017-11-17 | 天津大学 | Depth map super resolution ratio reconstruction method based on convolutional neural networks |
| CN109741260A (en) * | 2018-12-29 | 2019-05-10 | 天津大学 | A kind of efficient super-resolution method based on depth back projection network |
-
2020
- 2020-10-26 CN CN202011153555.1A patent/CN112365554A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107358576A (en) * | 2017-06-24 | 2017-11-17 | 天津大学 | Depth map super resolution ratio reconstruction method based on convolutional neural networks |
| CN109741260A (en) * | 2018-12-29 | 2019-05-10 | 天津大学 | A kind of efficient super-resolution method based on depth back projection network |
Non-Patent Citations (1)
| Title |
|---|
| RENHE LIU 等: "AN END-TO-END MULTI-SCALE RESIDUAL RECONSTRUCTION NETWORK FOR IMAGE COMPRESSIVE SENSING", 《ICIP 2019》 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113077044A (en) * | 2021-03-18 | 2021-07-06 | 北京工业大学 | General lossless compression and acceleration method for convolutional neural network |
| CN113034414B (en) * | 2021-03-22 | 2022-11-11 | 上海交通大学 | Image reconstruction method, system, device and storage medium |
| CN113034414A (en) * | 2021-03-22 | 2021-06-25 | 上海交通大学 | Image reconstruction method, system, device and storage medium |
| CN113658282A (en) * | 2021-06-25 | 2021-11-16 | 陕西尚品信息科技有限公司 | Image compression and decompression method and device |
| CN113688747A (en) * | 2021-08-27 | 2021-11-23 | 国网浙江省电力有限公司双创中心 | Method, system, device and storage medium for detecting personnel target in image |
| CN113688747B (en) * | 2021-08-27 | 2024-04-09 | 国网浙江省电力有限公司双创中心 | Method, system, device and storage medium for detecting personnel target in image |
| CN114066825A (en) * | 2021-10-29 | 2022-02-18 | 浙江工商大学 | Improved complex texture image flaw detection method based on deep learning |
| CN114066825B (en) * | 2021-10-29 | 2024-05-28 | 浙江工商大学 | Improved complex texture image flaw detection method based on deep learning |
| CN114494569A (en) * | 2022-01-27 | 2022-05-13 | 光线云(杭州)科技有限公司 | Cloud rendering method and device based on lightweight neural network and residual streaming transmission |
| CN114494569B (en) * | 2022-01-27 | 2023-09-19 | 光线云(杭州)科技有限公司 | Cloud rendering method and device based on lightweight neural network and residual streaming |
| CN115330635A (en) * | 2022-08-25 | 2022-11-11 | 苏州大学 | Image compression artifact removing method and device and storage medium |
| CN115330635B (en) * | 2022-08-25 | 2023-08-15 | 苏州大学 | Image compression artifact removing method, device and storage medium |
| CN116468812A (en) * | 2023-05-16 | 2023-07-21 | 山东省计算中心（国家超级计算济南中心） | Image compressed sensing reconstruction method and system based on multiple branches and multiple scales |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112365554A (en) | Compressed sensing image reconstruction method based on multi-scale residual error neural network | |
| Guo et al. | Deep wavelet prediction for image super-resolution | |
| Wang et al. | Ultra-dense GAN for satellite imagery super-resolution | |
| CN111932461B (en) | Self-learning image super-resolution reconstruction method and system based on convolutional neural network | |
| Chen et al. | Content-aware scalable deep compressed sensing | |
| Zhu et al. | Efficient single image super-resolution via hybrid residual feature learning with compact back-projection network | |
| CN104199627B (en) | Gradable video encoding system based on multiple dimensioned online dictionary learning | |
| CN107123094B (en) | Video denoising method mixing Poisson, Gaussian and impulse noise | |
| CN110111256A (en) | Image Super-resolution Reconstruction method based on residual error distillation network | |
| CN109949217B (en) | Video super-resolution reconstruction method based on residual learning and implicit motion compensation | |
| Yang et al. | Image super-resolution based on deep neural network of multiple attention mechanism | |
| CN116168067B (en) | Supervised multi-modal light field depth estimation method based on deep learning | |
| CN112950480A (en) | Super-resolution reconstruction method integrating multiple receptive fields and dense residual attention | |
| Zhao et al. | Deep equilibrium models for video snapshot compressive imaging | |
| CN109146785A (en) | A kind of image super-resolution method based on the sparse autocoder of improvement | |
| CN109672885B (en) | Video image coding and decoding method for intelligent monitoring of mine | |
| Zeng et al. | Image reconstruction of IoT based on parallel CNN | |
| CN110288529A (en) | A kind of single image super resolution ratio reconstruction method being locally synthesized network based on recurrence | |
| Liu et al. | An end-to-end multi-scale residual reconstruction network for image compressive sensing | |
| CN113284202A (en) | Image compression sensing method of scalable network based on content self-adaption | |
| Xu et al. | Fast and accurate image super-resolution using a combined loss | |
| Zhang et al. | Image super-resolution via RL-CSC: when residual learning meets convolutional sparse coding | |
| Qin et al. | Remote sensing image super-resolution using multi-scale convolutional neural network | |
| Cai et al. | Deep Sampling Networks | |
| CN106534861B (en) | Compression of images and decompressing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210212 |