CN114581539A - Compressed sensing image reconstruction method, device, storage medium and system - Google Patents

Compressed sensing image reconstruction method, device, storage medium and system Download PDF

Info

Publication number
CN114581539A
CN114581539A CN202210092192.8A CN202210092192A CN114581539A CN 114581539 A CN114581539 A CN 114581539A CN 202210092192 A CN202210092192 A CN 202210092192A CN 114581539 A CN114581539 A CN 114581539A
Authority
CN
China
Prior art keywords
network
sub
image
reconstruction
reconstructed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210092192.8A
Other languages
Chinese (zh)
Inventor
朱冬
张建
王杰
宋雯
唐国梅
杨易
张静
周宇杰
仲元红
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chongqing Qiteng Technology Co ltd
Original Assignee
Chongqing Qiteng Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chongqing Qiteng Technology Co ltd filed Critical Chongqing Qiteng Technology Co ltd
Priority to CN202210092192.8A priority Critical patent/CN114581539A/en
Publication of CN114581539A publication Critical patent/CN114581539A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/002—Image coding using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G06N3/065—Analogue means

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Neurology (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a compressed sensing image reconstruction method, a device, a storage medium and a system. The method comprises the following steps: acquiring image data to be reconstructed, and inputting the image data to be reconstructed into a pre-trained image reconstruction model to obtain a reconstructed image; the image reconstruction model carries out multiple times of iterative processing on image data to be reconstructed by utilizing a semi-quadratic splitting depth reconstruction network to obtain a reconstructed image; the semi-quadratic splitting depth reconstruction network comprises an x sub-problem network, a b sub-problem network and a q sub-problem network which are sequentially connected, wherein the x sub-problem network comprises a multilayer convolutional neural network; b, the sub-problem network comprises a soft threshold module; the q sub-problem network comprises a non-local neural network. The complex image reconstruction problem is decomposed into 3 simple sub-optimization problems of an x sub-problem, a b sub-problem and a q sub-problem, the image reconstruction efficiency is improved, and the image is reconstructed by combining sparsity and non-local prior, so that the image has higher visual quality, the image texture is clearer, and the reconstructed image is more accurate.

Description

Compressed sensing image reconstruction method, device, storage medium and system
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method, an apparatus, a storage medium, and a system for reconstructing a compressed sensing image.
Background
The Compressed Sensing (CS) theory breaks the constraint of the conventional nyquist sampling theorem, and does not put forward the requirement of twice the highest frequency for the sampling frequency any more, which indicates that, on the premise that the signal has sparsity, the compression and sampling of the signal are directly realized by mapping the high-dimensional signal to the low-dimensional signal, and finally, the accurate reconstruction of the original signal can be realized by solving the nonlinear optimization problem. The compressed sensing theory combines compression and sampling, and quickly becomes a research hotspot in the fields of image video processing, radar, medical nuclear magnetic resonance and the like after self-extraction.
The reconstruction problem is a key point of compressed sensing theory research, and is mainly realized by solving an NP-Hard problem to realize accurate reconstruction of signals, but the problem has no unique solution, so that researchers develop a great deal of research on the solution of the reconstruction problem. For many years, research on compressed sensing reconstruction algorithms has largely been divided into two categories: one is a conventional reconstruction algorithm and the other is a deep learning based reconstruction algorithm.
The traditional reconstruction algorithm is the most common reconstruction method except for the earlier non-convex optimization algorithm, greedy matching pursuit algorithm and convex optimization algorithm. The first three reconstruction algorithms are developed on the basis of signal sparsity, but the recovery effect is usually not satisfactory. The reconstruction algorithm based on the model is researched by surrounding the prior information of the signal, namely different reconstruction models are established by utilizing different prior information, and the reconstruction effect of the reconstruction algorithm is superior to that of other three algorithms. However, the conventional reconstruction algorithm needs to consume time on image block search or matrix inversion operation, which obviously increases the calculation time and reduces the algorithm efficiency.
The deep learning has been widely advocated and researched by researchers in various fields because of the excellent performance of the deep learning in various advanced image processing tasks. In recent years, researchers have proposed a deep learning framework to be applied to the field of compressed sensing, which is considered to be accepted by many researchers once being proposed, and have also developed many excellent depth compressed sensing reconstruction algorithms, namely, the Stacked Denoising auto encoder (SDA) proposed by Ali mouswavi, which implements end-to-end mapping between measured values and real values in an unsupervised manner. The non-iterative reconstruction network (Reconnet) uses the convolutional neural network in compressed sensing for the first time, and realizes effective reconstruction of images through a full connection layer and a convolutional layer. A Deep Residual error Network (DR 2-Net) proposed later on the basis of Reconnet combines the structure of Reconnet and the idea of Residual error, and further improves the quality of reconstructed images. Although the existing image reconstruction algorithm based on deep learning effectively overcomes the defects of the traditional compressed sensing algorithm to a certain extent, the problem of blocking effect still exists (the transformation coding based on the block is widely applied to image compression coding, the quantization becomes rough along with the reduction of the code rate, the discontinuity can occur at the boundary of the block, the obvious defect of forming a reconstructed image is called blocking effect), and the prior information of the image is not well utilized.
Disclosure of Invention
The invention aims to at least solve the technical problems in the prior art, and particularly innovatively provides a compressed sensing image reconstruction method, device, storage medium and system.
In order to achieve the above object of the present invention, according to a first aspect of the present invention, there is provided a compressed sensing image reconstruction method including: acquiring image data to be reconstructed, and inputting the image data to be reconstructed into a pre-trained image reconstruction model to obtain a reconstructed image; the image reconstruction model comprises a semi-quadratic splitting depth reconstruction network, and the image reconstruction model performs multiple iterative processing on image data to be reconstructed by using the semi-quadratic splitting depth reconstruction network to obtain a reconstructed image; the semi-quadratic splitting depth reconstruction network comprises an x sub-problem network, a b sub-problem network and a q sub-problem network which are sequentially connected, wherein the x sub-problem network comprises a multilayer convolutional neural network; the b sub-problem network comprises a soft threshold module; the q sub-problem network comprises a non-local neural network.
The technical scheme is as follows: the complex image reconstruction problem is decomposed into 3 simple sub-optimization problems of an x sub-problem, a b sub-problem and a q sub-problem by adopting semi-quadratic splitting to solve, the solving process is simplified, the image reconstruction efficiency is improved, and the three sub-problems combine sparsity prior and non-local prior to reconstruct the image, so that the reconstruction accuracy is improved; in addition, the image reconstruction model utilizes the semi-quadratic splitting depth reconstruction network to carry out repeated iteration processing on the image data to be reconstructed to obtain a reconstructed image, and the reconstruction accuracy is further improved.
In a preferred embodiment of the present invention, the x sub-problem network comprises 5 layers of convolutional neural networks connected in sequence; the first layer of convolutional neural network comprises a first x subproblem convolutional layer and a first x subproblem activation function layer, and the convolutional kernel size of the first x subproblem convolutional layer is 3 multiplied by 1; the second layer of convolutional neural network, the third convolutional neural network and the fourth convolutional neural network comprise a second x subproblem convolutional layer and a second x subproblem activation function layer, and the size of a convolutional core of the second x subproblem convolutional layer is 3 x 32; the fifth convolutional neural network includes a fifth x sub-problem convolutional layer and a fifth x sub-problem activation function layer, and the convolutional kernel size of the fifth x sub-problem convolutional layer is 3 × 3 × 32.
The technical scheme is as follows: based on the strong approximation capability of the neural network, the shallow convolutional neural network is used for approximating the problem of x, the relation between the degraded image and the original image is learned through the convolutional neural network, the sparse prior is fully utilized, and the reconstruction speed and the reconstruction precision can be accelerated.
In a preferred embodiment of the present invention, the b-sub-problem network includes a first b-sub-problem convolution layer, a second b-sub-problem convolution layer, a first b-sub-problem activation function layer, a third b-sub-problem convolution layer, a soft threshold module, a fourth b-sub-problem convolution layer, a second b-sub-problem activation function layer, a fifth b-sub-problem convolution layer, and a sixth b-sub-problem convolution layer, which are connected in sequence.
The technical scheme is as follows: the b sub-problem network has a simple structure, makes full use of the sparse prior of the image, and can realize high-efficiency processing.
In a preferred embodiment of the present invention, the q-sub-problem network comprises a first q-sub-problem convolutional layer, a non-local neural network and a second q-sub-problem convolutional layer connected in sequence.
The technical scheme is as follows: the non-local similarity prior of the image is fully utilized, and efficient and accurate image reconstruction can be realized.
In a preferred embodiment of the present invention, the image reconstruction model performs iterative processing on image data to be reconstructed by using the semi-quadratic splitting depth reconstruction network for a preset number of times, where the preset number of times is 6 to 12, to obtain a reconstructed image.
The technical scheme is as follows: and the accuracy of the reconstructed image is increased by multiple times of iteration processing, and the preset times are within the range of the optimal iteration times selected in the preliminary test, so that the proper calculation amount is ensured on the premise of increasing the accuracy of the reconstructed image.
In a preferred embodiment of the present invention, the image reconstruction model further includes an initial reconstruction network disposed before the semi-quadratic splitting depth reconstruction network, and the initial reconstruction network reconstructs a transpose matrix of the sampling matrix into a plurality of filters and reconstructs tensors output by the filters to obtain an initial reconstruction image.
The technical scheme is as follows: compared with the existing deep learning-based method, the method has the advantages that the image features can be amplified through the initial reconstruction network, and the number of parameters of the network is obviously reduced.
In a preferred embodiment of the present invention, the image reconstruction model training process includes: constructing a sampling network, an initial reconstruction network and a semi-quadratic splitting depth reconstruction network, wherein the sampling network is used for sampling an original image; acquiring a plurality of original images to construct a training set; and performing combined training on the sampling matrix, the initial reconstruction network and the semi-quadratic splitting depth reconstruction network based on a training set until a preset condition is reached, wherein the preset condition is that the training times reach a preset target time or the numerical value of a loss function is less than or equal to a preset loss threshold value.
The technical scheme is as follows: sampling is carried out through a sampling network, and the sampling network, the initial reconstruction network and the semi-quadratic splitting depth reconstruction network are trained jointly, so that the training of a plurality of image blocks is facilitated.
In order to achieve the above object of the present invention, according to a second aspect of the present invention, there is provided an image restoration apparatus comprising: the acquisition module acquires image data to be reconstructed; the image reconstruction model module comprises a semi-quadratic split depth reconstruction network, and the image reconstruction model module performs multiple iterative processing on image data to be reconstructed by using the semi-quadratic split depth reconstruction network to obtain a reconstructed image; the semi-quadratic splitting depth reconstruction network comprises an x sub-problem network, a b sub-problem network and a q sub-problem network which are sequentially connected, wherein the x sub-problem network comprises a multilayer convolutional neural network; the b sub-question network comprises a soft threshold module; the q sub-problem network comprises a non-local neural network.
The technical scheme is as follows: the complex image reconstruction problem is decomposed into 3 simple sub-optimization problems of an x sub-problem, a b sub-problem and a q sub-problem by adopting semi-quadratic splitting to solve, the solving process is simplified, the image reconstruction efficiency is improved, and the three sub-problems combine sparsity prior and non-local prior to reconstruct the image, so that the reconstruction accuracy is improved; in addition, the image reconstruction model utilizes the semi-quadratic splitting depth reconstruction network to carry out repeated iteration processing on the image data to be reconstructed to obtain a reconstructed image, and the reconstruction accuracy is further improved.
To achieve the above object, according to a third aspect of the present invention, there is provided a computer-readable storage medium having stored therein computer-executable instructions for implementing the method according to the first aspect of the present invention when executed by a processor.
The technical scheme is as follows: the complex image reconstruction problem is decomposed into 3 simple sub-optimization problems of an x sub-problem, a b sub-problem and a q sub-problem by adopting semi-quadratic splitting to solve, the solving process is simplified, the image reconstruction efficiency is improved, and the three sub-problems combine sparsity prior and non-local prior to reconstruct the image, so that the reconstruction accuracy is improved; in addition, the image reconstruction model utilizes the semi-quadratic splitting depth reconstruction network to carry out repeated iteration processing on the image data to be reconstructed to obtain a reconstructed image, and the reconstruction accuracy is further improved.
To achieve the above object, according to a fourth aspect of the present invention, there is provided an image compressive sensing system comprising an image sampling device and an image restoration device according to the second aspect of the present invention, wherein the image sampling device comprises a sampling network module, and the sampling network module is jointly trained with a semi-quadratic splitting depth reconstruction network in the image restoration device.
The technical scheme is as follows: the system jointly trains a sampling network, an initial reconstruction network and a semi-quadratic splitting depth reconstruction network, firstly utilizes a convolutional layer analog sampling process, and is more beneficial to training a plurality of image blocks compared with a traditional sampling mode. Initial reconstruction is then performed using convolutional layers and pixel shuffling, significantly reducing the number of parameters compared to existing deep learning-based methods. The system establishes a reconstruction model in a depth image compressed sensing network by combining image sparse prior and non-local similarity prior, decomposes the reconstruction model into a plurality of sub-problems by utilizing semi-quadratic splitting, solves each sub-problem under a frame of depth learning, and has obvious quality improvement compared with the prior art.
Drawings
FIG. 1 is a flow chart of a compressed perceptual image reconstruction method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a compressed sensing system according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a sub-problem network structure according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a sub-problem network structure of b in an embodiment of the present invention;
FIG. 5 is a schematic diagram of a q-sub-problem network structure in accordance with an embodiment of the present invention;
FIG. 6 is a schematic diagram of a non-local neural network architecture in accordance with an embodiment of the present invention;
FIG. 7 is a comparison graph of a reconstructed image output by an image reconstruction model and an original image according to an embodiment of the present invention;
FIG. 8 is an iterative PANR curve with training data for three compression ratios for an image reconstruction model according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention and should not be construed as limiting the present invention.
In the description of the present invention, it is to be understood that the terms "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. indicate orientations or positional relationships based on those shown in the drawings, and are merely for convenience of description and simplification of description, but do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, are not to be construed as limiting the present invention.
In the description of the present invention, unless otherwise specified and limited, it is to be noted that the terms "mounted," "connected," and "connected" are to be interpreted broadly, and may be, for example, a mechanical connection or an electrical connection, a communication between two elements, a direct connection, or an indirect connection via an intermediate medium, and specific meanings of the terms may be understood by those skilled in the art according to specific situations.
The invention discloses a compressed sensing image reconstruction method, which comprises the following steps in a preferred embodiment as shown in figure 1:
in step S1, image data to be reconstructed is acquired. And the image data to be reconstructed is data obtained after the original image is compressed and sensed. To adapt to the computing power of the computer, a whole original image is usually divided into a plurality of image blocks, the size of the image block is B × B, B is a positive integer, and B is preferably, but not limited to 33. Let the block sampling matrix be AB∈RM×NM < N, R represents the real number domain. N denotes the square of the image block size, N ═ B2I.e. the dimensions representing the image blocks. And M is a positive integer and is the product of the sampling rate and N, and represents the dimensionality of the sampled compressed data. The block sampling matrix is preferably, but not limited to, a random gaussian matrix. And sampling the image blocks through the block sampling matrix to obtain compressed data corresponding to the image blocks, and transmitting the compressed data to a reconstruction device end to be used as image data to be reconstructed.
And step S2, inputting the image data to be reconstructed into a pre-trained image reconstruction model to obtain a reconstructed image. As shown in fig. 2, the image reconstruction model includes a half-quadratic splitting depth reconstruction network, and the image reconstruction model performs multiple iteration processing on the image data to be reconstructed by using the half-quadratic splitting depth reconstruction network to obtain a reconstructed image, that is, the image data to be reconstructed is input into the half-quadratic splitting depth reconstruction network for processing to obtain a first intermediate result, the first intermediate result is input into the half-quadratic splitting depth reconstruction network for processing to obtain a second intermediate result, and multiple iteration results are obtained by successive approximation and output as the reconstructed image. The semi-quadratic splitting depth reconstruction network comprises an x sub-problem network, a b sub-problem network and a q sub-problem network which are connected in sequence, wherein the x sub-problem network comprises a multilayer convolutional neural network; b, the sub-problem network comprises a soft threshold module; the q sub-problem network comprises a non-local neural network.
In the present embodiment, it is preferable that the image data to be reconstructed is image compressed data obtained by sampling a network. The sampling network is a convolution network, the sampling process is simulated by a layer of unbiased convolution network, the input of the sampling network is set as an image block with the size of BxB, and the block sampling matrix A is adoptedB∈RM×NReshaped into M learnable filters, each having a kernel size
Figure BDA0003489510240000081
The sampling process can be expressed as:
Figure BDA0003489510240000091
wherein x isi∈RNRepresenting the i-th image block, yi∈RMA measurement value (image compressed data) representing the ith image block, is a convolution operation,
Figure BDA0003489510240000092
for the filter weight matrix, the variation is learned at training.
In this embodiment, preferably, the image reconstruction model further includes an initial reconstruction network disposed before the semi-quadratic splitting depth reconstruction network, and the initial reconstruction network reconstructs a transposed matrix of the sampling matrix into a plurality of filters and reconstructs tensors output by the filters to obtain an initial reconstruction image.
In this embodiment, the initial reconstruction network uses one convolutional layer, and first, the block sampling matrix A is usedBTransposed matrix A ofB T∈RM×NReshaped into N filters, each filterThe kernel sizes are all 1 × 1 × M, and then the tensors with the size of N × 1 × 1 are reshaped into the size of N × 1 × 1 by pixel shuffling
Figure BDA0003489510240000093
As an initial reconstructed image, the initial reconstruction process can be expressed as:
Figure BDA0003489510240000094
wherein x isi 0An initial reconstructed image representing an ith original image block; PixelShuffle (-) represents a pixel shuffling operation, a common method of image and feature map enlargement in super resolution.
Figure BDA0003489510240000095
The convolutional layer weight matrix, which represents the initial reconstructed network, learns the changes during training.
In the embodiment, a sparse prior and a non-local similarity prior are combined to establish an optimized image reconstruction model as follows:
Figure BDA0003489510240000096
wherein x represents a reconstructed image; r (x) denotes the regularization term, i.e. a priori information, Rl1(x) Sparse positive rule of L1, Rnon-local(x) For non-local regularization, y represents the corresponding measured value of the original image. argmin represents taking the maximum value. The accurate reconstruction of the image can be realized by solving the above formula, but the direct solving of the above formula has high computational complexity, and in order to simplify the solving of the equation, the solution is performed by using Half Quadratic Splitting (HQS).
The process of introducing semi-quadratic splitting of a single variable includes: let the semi-quadratic split be aimed at the following optimization objectives:
Figure BDA0003489510240000101
wherein, the first term in the above formula is a data fidelity term, and the second term is a constraint term. First, introducing the variable z, and rewriting the above formula to obtain:
Figure BDA0003489510240000102
the cost function is:
Figure BDA0003489510240000103
where μ is a non-increasing penalty parameter. A is a sampling matrix from x to y.
And expanding the calculation of the model by an alternate solution mode, wherein the alternate solution expression is as follows:
Figure BDA0003489510240000104
therefore, for the reconstruction model established by the invention, a half-quadratic splitting algorithm is adopted, auxiliary variables b and q are introduced, and the optimized image reconstruction model can be equivalent to the following 3 subproblems to be alternately solved:
Figure BDA0003489510240000105
wherein k represents the number of iterations, and the value range is 6 to 12, preferably 9; x is the number ofiRepresenting an ith image block to be estimated; mu.s1Representing a first penalty parameter; mu.s2Representing a second penalty parameter; lambda [ alpha ]1A first penalty coefficient of 0.01; lambda [ alpha ]2A second penalty factor of 0.01; biRepresenting the input of the b sub-problem network when the ith image block is reconstructed;
Figure BDA0003489510240000106
indicating that 2 ranges are to be obtainedThe square of the number; rl1(bi) Denotes biAn L1 norm regularizer term which is input; q. q.siRepresenting the input of the q sub-problem network when the ith image block is reconstructed; rnon-local(qi) Denotes qiIs an input non-local regularization term.
Figure BDA0003489510240000111
Representing the value of an auxiliary variable b obtained in the k iteration in the reconstruction process of the ith image block;
Figure BDA0003489510240000112
representing the value of an auxiliary variable q obtained in the k iteration in the reconstruction process of the ith image block; if the mathematical solution mode is directly used, the problems of high calculation complexity and low model performance exist. The invention converts the solution of three equal equations in the above equation into the solution of three sub-problems: the first equation is an x subproblem, which is realized by a Convolutional Neural Network (CNN); the second equation is a b sub-problem, and the b sub-problem is solved in a learning mode; the third equation, the q sub-problem, is solved on a learnable non-local network basis.
For the x sub-problem, the expression is:
Figure BDA0003489510240000113
equation (1) is a convex problem, and can directly solve a closed solution, but the direct solution computation is very large. Because of the strong approximation capability of the neural network, the invention directly approximates the formula (1) by using a shallow CNN network. As can be seen from formula (1), xiIs the ith original image block to be estimated,
Figure BDA0003489510240000114
and
Figure BDA0003489510240000115
are degraded image blocks of the i-th original image block,
Figure BDA0003489510240000116
is the reconstructed image block corresponding to the ith original image block in the (k + 1) th iteration. In particular, the relation between a degraded image block and an original image block is learned using a convolutional neural network CNN, which solves the above problem by a CNN structure based on convolution (Conv) and activation function (ReLU), a specific network structure being shown in fig. 3.
As shown in fig. 3, the x sub-problem network includes 5 layers of convolutional neural networks connected in sequence; the first layer of convolutional neural network comprises a first x subproblem convolutional layer and a first x subproblem activation function layer, and the convolutional kernel size of the first x subproblem convolutional layer is 3 multiplied by 1; the second layer of convolutional neural network, the third convolutional neural network and the fourth convolutional neural network comprise a second x subproblem convolutional layer and a second x subproblem activation function layer, and the size of a convolutional core of the second x subproblem convolutional layer is 3 x 32; the fifth convolutional neural network includes a fifth x subproblem convolutional layer and a fifth x subproblem activation function layer, and the convolutional kernel size of the fifth x subproblem convolutional layer is 3 × 3 × 32.
As shown in fig. 3, the solution of the x sub-problem is a non-linear reconstruction process, and in the present invention, CNN is used to achieve this function. The network comprises 5 layers, wherein the network structure of the middle 3 layers is the same except for the first layer and the last layer. In the middle layer, the output of each layer is a feature map under 32 channels. The first layer is run on the initial reconstructed output, which has 32 signatures. The last layer consists of a single filter of 32 channels.
In this embodiment, the expression of the b sub-problem is:
Figure BDA0003489510240000121
for R in the above formula (2)l1(bi) Again using a general non-linear transformation function
Figure BDA0003489510240000122
Instead, the expression becomes:
Figure BDA0003489510240000123
there is a non-linear transformation such that:
Figure BDA0003489510240000124
wherein α is only with
Figure BDA0003489510240000125
The relevant parameter, let shrinkage threshold θ be λ1The value of θ may be taken to be 0.01. Thus, equation (3) can be written as:
Figure BDA0003489510240000126
Figure BDA0003489510240000127
a 1 norm representing a nonlinear transformation function; thus, the device
Figure BDA0003489510240000128
The closed-form solution of (c) is:
Figure BDA0003489510240000129
the solving expression for b is therefore:
Figure BDA00034895102400001210
the parameters are learnable in each iteration, so there are:
Figure BDA0003489510240000131
θka value of θ representing the kth iteration; for the solving process of the sub-problem b, the network structure is shown in fig. 4, and the sub-problem b network includes a first sub-problem b convolution layer, a second sub-problem b convolution layer, a first sub-problem b activation function layer, a third sub-problem b convolution layer, a soft threshold module, a fourth sub-problem b convolution layer, a second sub-problem b activation function layer, a fifth sub-problem b convolution layer and a sixth sub-problem b convolution layer, which are connected in sequence. The soft threshold module is preferably, but not limited to, a Softmax function module.
In the present embodiment, a Non-local neural network (Non-local Net) is used to solve the q-sub problem. The non-local neural network is based on the principle of a non-local mean method, and realizes non-local operation of calculating the response of a certain position as the weighted sum of the features of all the positions. The q subproblem network comprises a first q subproblem convolutional layer, a non-local neural network and a second q subproblem convolutional layer which are connected in sequence. Regarding the q-sub problem, it is regarded as a nonlinear optimization problem, and the reconstruction process of the signal is realized by a non-local neural network, and the network parameters are shown in fig. 6, where ξ, ψ, δ are convolution kernels with the size of 1 × 1, and × represents the tensor product. The input x and the output z remain the same dimension.
In the embodiment, in order to obtain better reconstruction accuracy and efficiency, the image reconstruction model performs iteration processing on the image data to be reconstructed for a preset number of times by using a semi-quadratic splitting depth reconstruction network to obtain a reconstructed image, wherein the preset number of times is 6 to 12. That is, as shown in fig. 2, the loop iterates to perform a semi-quadratic split depth reconstruction network, with iterations numbers of 6 to 12, preferably 9, which are obtained from multiple experiments.
In a preferred embodiment, the image reconstruction model training process includes: constructing a sampling network, an initial reconstruction network and a semi-quadratic splitting depth reconstruction network, wherein the sampling network is used for sampling an original image; acquiring a plurality of original images to construct a training set; and performing combined training on the sampling matrix, the initial reconstruction network and the half-second splitting depth reconstruction network based on the training set until a preset condition is reached, wherein the preset condition is that the training times reach a preset target time or the numerical value of a loss function is less than or equal to a preset loss threshold value.
In this embodiment, the ith image block x will be giveniAs input, i is a positive integer, training is carried out on a joint sampling network, an initial reconstruction network and a semi-quadratic splitting depth reconstruction network to generate xiN of (A)RSub-iterative reconstruction of results
Figure BDA0003489510240000141
Denotes xiAs output. Strive to reduce xiAnd
Figure BDA0003489510240000142
while satisfying the symmetry constraint
Figure BDA0003489510240000143
(symbol)
Figure BDA0003489510240000145
Are symmetrically constrained symbols. Therefore, the end-to-end loss function of the proposed algorithm of the present invention is designed as follows:
Figure BDA0003489510240000144
wherein N isPRepresenting the total number of image blocks participating in the training. N denotes each image block xiSize of (A) is B2。NRAnd the number of times of iteration processing of the preset semi-quadratic splitting depth reconstruction network is shown, and is preferably 9.γ represents a regularization parameter, and is set to 0.01.
The present invention also discloses an image restoration device, including: the acquisition module acquires image data to be reconstructed; the image reconstruction model module comprises a semi-quadratic splitting depth reconstruction network, and the image reconstruction model module performs multiple iterative processing on image data to be reconstructed by using the semi-quadratic splitting depth reconstruction network to obtain a reconstructed image; the semi-quadratic splitting depth reconstruction network comprises an x sub-problem network, a b sub-problem network and a q sub-problem network which are sequentially connected, wherein the x sub-problem network comprises a multilayer convolutional neural network; the b sub-problem network comprises a soft threshold module; the q sub-problem network comprises a non-local neural network.
The invention also discloses a computer readable storage medium, wherein a computer executing instruction is stored in the computer readable storage medium, and when a processor executes the computer executing instruction, the image reconstruction method provided by the invention is realized.
The invention also discloses an image compression sensing system, which comprises image sampling equipment and the image recovery equipment provided by the invention, wherein the image sampling equipment comprises a sampling network module, and the sampling network module and a semi-quadratic splitting depth reconstruction network in the image recovery equipment are jointly trained. The image sampling device is in communication connection with the image recovery device, the image sampling device samples an original image and transmits the sampled data to the image recovery device, and the image recovery device reconstructs the original image based on the obtained sampled data.
In an application scenario, the compressed sensing image reconstruction method provided by the invention is verified. To verify the effectiveness and image reconstruction quality of the network proposed herein, 88912 randomly cropped image blocks, each of which has a size of 33 × 33, were extracted using a 91image data set, i.e., a data set of 91 images, which is consistent with a plurality of mainstream depth learning-based reconstruction algorithms. For the test set, three widely used reference data sets were selected: set5, Set11, BSD68, the three reference datasets containing 5, 11, and 68 grayscale images, respectively.
All experiments in the application scene run under the calculation of an Intel i7-8700K processor, 3.2GHz main frequency and 15.6GiB internal memory, and are accelerated by using a GTX1080Ti GPU, wherein an operating system is Ubuntu 16.04 release. The evaluation criterion uses Peak Signal to Noise Ratio (PSNR) as an objective quality evaluation index.
In order to verify the effectiveness of the reconstruction network provided by the invention, three pictures of parrots, houses and butterflies are selected from the data set for testing. The original pictures are input into a sampling network to obtain compressed sampling data, wherein the parrot nautilus and the house pictures are compressed by adopting a compression ratio of 10%, and the butterfly pictures are compressed by adopting a compression ratio of 20%. And respectively inputting the compressed sampling data corresponding to the three original pictures into the image reconstruction model provided by the invention for processing to obtain a reconstructed image. As shown in fig. 7, the left column is the original image of the parrot, house, and butterfly from top to bottom, and the right column is the reconstructed image of the parrot, house, and butterfly from top to bottom. In fig. 7, the larger intra-square image in the lower right corner is an enlargement of the intra-square image in the small square in the figure. As can be seen from FIG. 7, the difference between the reconstructed picture and the original picture is small, PSNR of parrots, houses and butterflies respectively reaches 28.95dB, 33.02dB and 32.88dB, and the reconstruction method provided by the invention has high visual quality and very clear image texture. In summary, the compressed image reconstruction algorithm provided by the invention is an effective and accurate reconstruction algorithm.
Fig. 8 shows an iterative variation curve of the PSNR with the training data under different compression ratios, and it can be seen from fig. 8 that the PSNR value of the image reconstruction model provided by the present invention gradually becomes stable as the training times increase, i.e., the reconstruction algorithm provided by the present invention is a stable and effective reconstruction algorithm. MR stands for compression ratio.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims (10)

1. A method for compressed sensing image reconstruction, comprising:
acquiring image data to be reconstructed, and inputting the image data to be reconstructed into a pre-trained image reconstruction model to obtain a reconstructed image;
the image reconstruction model comprises a semi-quadratic splitting depth reconstruction network, and the image reconstruction model performs multiple iterative processing on image data to be reconstructed by using the semi-quadratic splitting depth reconstruction network to obtain a reconstructed image;
the semi-quadratic splitting depth reconstruction network comprises an x sub-problem network, a b sub-problem network and a q sub-problem network which are sequentially connected, wherein the x sub-problem network comprises a multilayer convolutional neural network; the b sub-problem network comprises a soft threshold module; the q sub-problem network comprises a non-local neural network.
2. The method of compressed sensing image reconstruction according to claim 1, wherein the x sub-problem network comprises sequentially connected 5 layers of convolutional neural networks;
the first layer of convolutional neural network comprises a first x subproblem convolutional layer and a first x subproblem activation function layer, and the convolutional kernel size of the first x subproblem convolutional layer is 3 multiplied by 1;
the second layer of convolutional neural network, the third convolutional neural network and the fourth convolutional neural network comprise a second x subproblem convolutional layer and a second x subproblem activation function layer, and the size of a convolutional core of the second x subproblem convolutional layer is 3 x 32;
the fifth convolutional neural network includes a fifth x subproblem convolutional layer and a fifth x subproblem activation function layer, and the convolutional kernel size of the fifth x subproblem convolutional layer is 3 × 3 × 32.
3. The method of claim 1, wherein the b-sub-problem network comprises a first b-sub-problem convolutional layer, a second b-sub-problem convolutional layer, a first b-sub-problem activation function layer, a third b-sub-problem convolutional layer, a soft threshold module, a fourth b-sub-problem convolutional layer, a second b-sub-problem activation function layer, a fifth b-sub-problem convolutional layer and a sixth b-sub-problem convolutional layer which are connected in sequence.
4. The method according to claim 1, wherein the q-sub-problem network comprises a first q-sub-problem convolutional layer, a non-local neural network and a second q-sub-problem convolutional layer which are connected in sequence.
5. The compressed sensing image reconstruction method according to one of claims 1 to 4, wherein the image reconstruction model performs a preset number of iterations on image data to be reconstructed by using the semi-quadratic splitting depth reconstruction network to obtain a reconstructed image, and the preset number is 6 to 12.
6. The method according to claim 5, wherein the image reconstruction model further includes an initial reconstruction network disposed before the semi-quadratic splitting depth reconstruction network, and the initial reconstruction network reconstructs a transpose of a sampling matrix into a plurality of filters and reconstructs tensors output by the filters to obtain an initial reconstruction image.
7. The method of compressed sensing image reconstruction according to claim 5, wherein the image reconstruction model training process comprises:
constructing a sampling network, an initial reconstruction network and a semi-quadratic splitting depth reconstruction network, wherein the sampling network is used for sampling an original image;
acquiring a plurality of original images to construct a training set;
and performing combined training on the sampling matrix, the initial reconstruction network and the semi-quadratic splitting depth reconstruction network based on a training set until a preset condition is reached, wherein the preset condition is that the training times reach a preset target time or the numerical value of a loss function is less than or equal to a preset loss threshold value.
8. An image restoration apparatus characterized by comprising:
the acquisition module acquires image data to be reconstructed;
the image reconstruction model module comprises a semi-quadratic split depth reconstruction network, and the image reconstruction model module performs multiple times of iterative processing on image data to be reconstructed by using the semi-quadratic split depth reconstruction network to obtain a reconstructed image; the semi-quadratic splitting depth reconstruction network comprises an x sub-problem network, a b sub-problem network and a q sub-problem network which are sequentially connected, wherein the x sub-problem network comprises a multilayer convolutional neural network; the b sub-problem network comprises a soft threshold module; the q sub-problem network comprises a non-local neural network.
9. A computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a processor, implement the method of any one of claims 1-7.
10. An image compressive sensing system comprising an image sampling device and the image restoration device of claim 8, the image sampling device comprising a sampling network module trained in conjunction with a semi-quadric split depth reconstruction network in the image restoration device.
CN202210092192.8A 2022-01-26 2022-01-26 Compressed sensing image reconstruction method, device, storage medium and system Pending CN114581539A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210092192.8A CN114581539A (en) 2022-01-26 2022-01-26 Compressed sensing image reconstruction method, device, storage medium and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210092192.8A CN114581539A (en) 2022-01-26 2022-01-26 Compressed sensing image reconstruction method, device, storage medium and system

Publications (1)

Publication Number Publication Date
CN114581539A true CN114581539A (en) 2022-06-03

Family

ID=81770937

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210092192.8A Pending CN114581539A (en) 2022-01-26 2022-01-26 Compressed sensing image reconstruction method, device, storage medium and system

Country Status (1)

Country Link
CN (1) CN114581539A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115063494A (en) * 2022-06-29 2022-09-16 北京航空航天大学 Mars image compression method and device, computer equipment and storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115063494A (en) * 2022-06-29 2022-09-16 北京航空航天大学 Mars image compression method and device, computer equipment and storage medium
CN115063494B (en) * 2022-06-29 2024-05-28 北京航空航天大学 Mars image compression method, device, computer equipment and storage medium

Similar Documents

Publication Publication Date Title
CN112884851B (en) Construction method of deep compressed sensing network based on expansion iteration optimization algorithm
CN113516601B (en) Image recovery method based on deep convolutional neural network and compressed sensing
CN111080567A (en) Remote sensing image fusion method and system based on multi-scale dynamic convolution neural network
CN113362250B (en) Image denoising method and system based on dual-tree quaternary wavelet and deep learning
CN110827198B (en) Multi-camera panoramic image construction method based on compressed sensing and super-resolution reconstruction
CN110288526B (en) Optimization method for improving imaging quality of single-pixel camera by image reconstruction algorithm based on deep learning
CN114820341A (en) Image blind denoising method and system based on enhanced transform
CN109741407A (en) A kind of high quality reconstructing method of the spectrum imaging system based on convolutional neural networks
CN113674172B (en) Image processing method, system, device and storage medium
CN112183742A (en) Neural network hybrid quantization method based on progressive quantization and Hessian information
CN108734675A (en) Image recovery method based on mixing sparse prior model
CN104408697B (en) Image Super-resolution Reconstruction method based on genetic algorithm and canonical prior model
CN111127490A (en) Medical image segmentation method based on cyclic residual U-Net network
CN111461978A (en) Attention mechanism-based resolution-by-resolution enhanced image super-resolution restoration method
CN114331913B (en) Motion blurred image restoration method based on residual attention block
CN115880158A (en) Blind image super-resolution reconstruction method and system based on variational self-coding
CN116797461A (en) Binocular image super-resolution reconstruction method based on multistage attention-strengthening mechanism
CN116757955A (en) Multi-fusion comparison network based on full-dimensional dynamic convolution
CN116797456A (en) Image super-resolution reconstruction method, system, device and storage medium
CN114581539A (en) Compressed sensing image reconstruction method, device, storage medium and system
Wen et al. The power of complementary regularizers: Image recovery via transform learning and low-rank modeling
CN117726540A (en) Image denoising method for enhanced gate control converter
CN113436198A (en) Remote sensing image semantic segmentation method for collaborative image super-resolution reconstruction
CN108596831B (en) Super-resolution reconstruction method based on AdaBoost example regression
CN116309221A (en) Method for constructing multispectral image fusion model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination