Fully-Convolutional Measurement Network for
Compressive Sensing Image Reconstruction

arXiv:1712.01641v1 [cs.CV] 21 Nov 2017

Xuemei Xie∗, Jiang Du, Chenye Wang, Guangming Shi, Xun Xu, Yuxiang
Wang
School of Electronic Engineering, Xidian University, Xi’an, Shaanxi 710071, PR China

Abstract
Recently, deep learning methods have made a significant improvement in compressive sensing image reconstruction task. However, it still remains a problem
of block effect which degrades the reconstruction results. In this paper, we propose a fully-convolutional network, where the full image is directly measured
with a convolutional layer. Thanks to the overlapped convolutional measurement, the block effect is removed. In addition, because of the jointly training
of the measurement and reconstruction stages, the adaptive measurement can
be obtained. Furthermore, to enhance the performance of the network, residual learning is used in the reconstruction network. Experimental results show
that the proposed method outperforms the existing methods in both PSNR and
visual effect.
Keywords: compressive sensing, fully-convolutional network, adaptive
measurement, block effect, residual network.

1. Introduction
Compressive sensing (CS) theory [1, 2, 3, 4] is able to acquire measurements
of signals at sub-Nyquist rates and recover signals with high probability when
the signals are sparse in a certain domain. Greedy algorithms [5, 6], convex
optimization algorithms [7, 8], and iterative algorithms [9, 10] have been used
∗ Corresponding

author
Email address: xmxie@mail.xidian.edu.cn (Xuemei Xie)

Preprint submitted to Neurocomputing

December 6, 2017

for recovering images in conventional CS. However, almost all these methods
recover images by solving an optimization problem, which is time-consuming.
In order to reduce the computational complexity in the reconstruction stage,
convolutional neural networks (CNNs) are applied to replace the optimization
process. CNN-based methods [11, 12, 13, 14, 15] use big data [16] to train the
networks that speed up the reconstruction stage. Mousavi, Patel, and Baraniuk
[11] firstly propose deep learning approach to solve the CS recovery problem.
They use stacked denoising autoencoders (SDA) to recover signals from undersampled measurements. ReconNet [12] and DeepInverse [13] introduce CNNs to
the reconstruction problem, where random Gaussian measurement matrix are
used to generate the measurements. Instead, the methods [14, 15] using adaptive measurement learn a transformation from signals to the measurements.
This mechanism allows measurements to retain more information from images.
The methods mentioned above divide an image into blocks, which breaks the
structure information of the image. This will cause the block effect in the reconstructed image.
In this paper, we propose a fully-convolutional network for CS image reconstruction. Instead of block-wise methods, a convolutional layer is applied to
obtain the measurement from a full image, which keeps the integrity of structure information of the original image. Furthermore, motivated by the residual
learning proposed by ResNet [17], we apply residual connection block (Resblock)
in the proposed network for improvement. Experimental results show that the
proposed method outperforms the state-of-the-art method 1∼2dB in PSNR at
different measurement rates.
The organization of this paper is as follows. The related works using deep
learning methods for the CS reconstruction problem are introduced in Section
2. Section 3 presents the proposed fully-convolutional network. Section 4 shows
experimental results of the proposed method and the previous works. The
conclusion of this paper is drawn in Section 5.

2

2. Related Work
Recently, deep learning methods have been applied in CS image reconstruction tasks and achieve promising results such as [11, 12, 14]. Among them,
CNN-based methods present superior performance. ReconNet [12] is a representative work that applies CNNs in reconstructing low-resolution mixed image
measured by random Gaussian matrix. The framework is shown in Fig. 1.

Figure 1: Framework of random Gaussian based network.

The training of the network is driven by the error between the label and the
reconstructed image. And the loss function is given by
L({W }) =

T
1X
2
kf (yi , {W }) − xi k ,
T i=1

(1)

where f (yi , {W }) is the i − th reconstructed image of ReconNet. xi is the i − th
original signal as well as the i − th label. {W } means the training parameters in
ReconNet. T is the total number of image blocks in the training dataset. The
loss function is minimized by tuning {W } using back propagation.
Based on the way the original image is measured, deep learning methods for
CS reconstruction can be divided into two categories: fixed random Gaussian
measurement and adaptive measurement.
Fixed random Gaussian measurement. Mousavi et al. [11] firstly use SDA to
recover signals from undersampled measurements. ReconNet [12] and DeepIn3

verse [13] utilizes CNNs to recover signals from Gaussian measurements. DR2 Net [18], inheriting ReconNet, adds residual connection blocks (Resblock) to
its reconstruction stage and achieves better performance. Instead of learning
to directly reconstruct the high-resolution image from the low-resolution one,
DR2 -Net learns the residual between the ground truth image and the preliminary reconstructed image. However, the measurements of these methods are
randomly measured, which is not optimally designed for natural images.
Adaptive measurement. In order to keep the information of the training data,
the adaptive measurement is proposed. Methods including improved ReconNet
[19], Adp-Rec1 [15], and DeepCodec [14] are all based on adaptive measurement. In cases of the improved ReconNet and Adp-Rec, a fully-connected layer
is used to measure the signals, which allows for a jointly learning of the measurement and reconstruction stages. With the learned measurement matrix, a
significant gain in terms of PSNR is achieved. DeepCodec, closely related to
the DeepInverse, learns a transformation for dimensionality reduction. Learning measurements from the original signals helps to preserve more information
compared with taking random measurements.

3. Fully-Convolutional Measurement
The exsiting CNN-based CS methods always adopt block-wise pattern due
to the limitation of GPU memory. The block effect comes accordingly. In order to overcome this shortcoming, we propose a fully-convolutional network in
which a convolutional layer is used to get the adaptive measurements. It is different from our previous work using fully-connected layers [15]. Fig. 2 shows the
proposed network which is composed of a convolutional layer, a deconvolutional
layer [20], and a residual block. The first layer ‘conv’ is used to obtain measurements. The second layer ‘deconv’ is used to recover a low resolution image
1 Adp-Rec

stands for adaptive measurement network for CS image reconstruction, proposed

in our previous work.

4

from the measurements. Furthermore, we apply a residual network (ResNet) to
reconstruct the high resolution image. Because batch normalization would get
rid of range flexibility from networks [21], we remove the batch normalization
layer in Resblock.
Back propagation
Resblock

conv

deconv

e
r
r
o
r

+
reconstruction

label
low-resolution

residual

Figure 2: Framework of the proposed network.

The loss function of the proposed network is given by
L({W }) =

T
1X
2
kf (xi , {W, K}) − xi k ,
T i=1

(2)

where K is the parameter of the convolutional measurement network, and W is
the parameters of the reconstruction network. The Euclidian distance between
the label and the reconstruction is back propagated to train the whole network.
The main advantage of the proposed network is the use of convolutional
layer as the measurement matrix. By means of the overlapped convolutional
kernels, this structure can remove block effect of the reconstructed images. In
details, one feature map contains several measurements of each pixel, which is
similar to the idea proposed by He et al. [22] that the feature map preserves the
explicit per-pixel spatial correspondence. Another advantage is that the fullyconvolutional neural network can deal with images of any size, which breaks the
limitation that fully-connected layer is only capable of measuring the fixed size
of images. Once the network is trained, we can test images with different sizes
5

without changing the structure of the network.
Fig. 3 shows an example of reconstruction results at three kinds of measurements. The original image and those by random Gaussian, Adp-Rec, and
the proposed method are shown respectively in Fig. 3(a), (b), (c), and (d).
The measurement rate is 10%. We can see that the proposed method performs
better than the others.

(a) Origin

(b) ReconNet

(c)

(21.49dB)

Adp-Rec
(26.65dB)

(d) Proposed
(27.61dB)

Figure 3: The reconstruction results of monarch at measurement rate 10%.

The better performance can be proved through a visualization of the kernels
in convolutional layer of the measurement network, as shown in Fig. 4. Since
the original signal in random Gaussian and adaptive measurements is a cloumn
vector (Fig. 4(a) and (b)), we reshape the row vectors of measurement matrix to
size 33 × 33. Fig. 4(a) shows two reshaped row vectors of the random Gaussian
measurement matrix at measurement rates 1% , 10%, and 25% in both time
and frequency domain. The content of random Gaussian measurement matrix
is obviously irregular. Fig. 4(b) shows two reshaped row vectors of adaptive
measurement matrix in Adp-Rec. When measurement rate is set to 1%, low
frequency information is already extracted. As the measurement rate increases,
much high frequency information is captured. Fig. 4(c) shows two kernels of the
proposed measurement matrix. Compared with the adaptive measurements in
Adp-Rec, the measurements by the proposed method provide more concentrated
energy in the low frequency area at different measurement rates. As for the
directional information, when measurement rate is 1%, two extracted typical
directions ‘horizontal’ and ‘vertical’ can be easily observed in time domain.
6

Time
domain

Frequency
domain
1%

10%

25%

(a) Random Gaussian measurements.

Time
domain

Frequency
domain
10%

1%

25%

(b) Adaptive measurements in Adp-Rec.

Time
domain

Frequency
domain

1%

10%

25%

(c) Proposed
Figure 4: Reshaped row vectors of measurement matrix at measurement rates 1%, 10%, and
25% in both time and frequency domain.

Fig. 5 shows the reconstruction of image ‘Monarch’, its low-resolution, and
the corresponding residual. From residual image in frequency domain, we can
see that the high frequency component is mainly learned by the residual network.
Rather than ReconNet which reconstructs the high resolution image from the
7

Reconstruction

Low-resolution

Residual

Time
domain

Frequency
domain

Figure 5: Reconstruction image, low-resolution image and residual image at measurement rate
10% in both time and frequency domain.

low resolution one directly, ResNet just reconstruct the residual between the
low resolution image and the high resolution image, that is the reconstruction
image. Thus, all its energy is concentrated on the residual. That is why ResNet
has better performance.

4. Experiments
In this section, we perform experiments on the reconstruction of compressive
sensing images with existing typical methods. The results show the outstanding
performance by the proposed method.
The experiments are conducted on caffe framework [23]. Our computer is
equipped with Intel Core i7-6700 CPU with frequency of 3.4GHz, 4 NVidia
GeForce GTX Titan XP GPUs, 128 GB RAM, and the framework runs on
Ubuntu 16.04 operating system. The training dataset consists of 800 pieces of
256 × 256 size images downsampled and divided from 800 images in DIV2K
dataset [24].
The performance of the proposed method is compared with those by Re-

8

conNet and Adp-Rec which are the typical CNN-based CS methods. We give
the testing results using image ‘parrots’, ‘flinstones’, and ‘cameraman’ at measurement rates 1%, 10%, and 25%, as shown in Fig. 6, Fig. 7, and Fig. 8,
respectively. The proposed method provides the best reconstruction results in
terms of PSNR and the results are most visually attractive.
From the results shown in Fig. 6, with measurement rate being 1%, it can
be seen that the block effect is removed (Fig 6(d) vs. (b) and (c)). From Fig. 7,
when the measurement rate is 10%, the proposed method shows the advantage
in reconstructing the image, typically in those smooth areas such as nose, hands,
and legs of the man. From Fig. 8, when measurement rate rises to 25%, the
proposed method still outperforms other methods, which can be easily seen in
the edge of the man’s arm.

(a) Origin

(b) ReconNet

(c)

(18.93dB)

Adp-Rec
(21.67dB)

(d) Proposed
(22.49dB)

Figure 6: The reconstruction results of parrots at measurement rate 1%.

(a) Origin

(b) ReconNet

(c)

(19.04dB)

Adp-Rec
(23.83dB)

(d) Proposed
(24.98dB)

Figure 7: The reconstruction results of flinstones at measurement rate 10%.

9

(a) Origin

(b) ReconNet

(c)

(23.48dB)

Adp-Rec
(27.11dB)

(d) Proposed
(28.99dB)

Figure 8: The reconstruction results of cameraman at measurement rate 25%.

For an overall look on the performance, the reconstruction results of 11 test
images at measurement rates 1%, 10%, and 25% with the methods including
ReconNet, DR2 -Net, Adp-Rec, Fully-Conv2 , and the proposed one are shown in
Table 1. The mean PSNR is given in the type of blue background. It is obvious
that the proposed method shows greatest performance in almost all test images.
From Table 1, it can be concluded that Adp-Rec beats DR2 -Net and ReconNet about 3dB in all measurement rates because of its adaptive measurement.
Based on the standard ReconNet [12], the improved ReconNet [19] adds several
tricks such as adaptive measurement and adversarial loss. Its performance is
even lower than Adp-Rec. Despite its promising results, Adp-Rec still divides
image into blocks, ignoring the relevance between neighbouring blocks, which
causes to the block effect in reconstructed images. For this reason, Fully-Conv
uses a fully-convolutional layer as measurement matrix to deal with this problem. It achieves comparable results with Adp-Rec even though it contains no
additional operation.
To further improve the reconstruction results, we put Resblock after FullyConv structure because of the brilliant performance of Resblock in reconstruction task. With this enhancement, the proposed method obtains the best performance in terms of PSNR at all measurement rates, as shown in Table 1.
2 Fully-Conv

consists of a convolutional layer and a deconvolutional layer without Resblock,

which can be regarded as the tiny model of the proposed network.

10

Table 1: The PSNR results at measurement rates (MR) 1%, 10%, and 25%, where Red is
ranked the first and blue is ranked the second.
MR

1%

10%

25%

Samples
Monarch
Parrots
Barbara
Boats
Cameraman
Fingerprint
Flinstones
Foreman
House
Lena
Peppers
Mean(all)
Monarch
Parrots
Barbara
Boats
Cameraman
Fingerprint
Flinstones
Foreman
House
Lena
Peppers
Mean(all)
Monarch
Parrots
Barbara
Boats
Cameraman
Fingerprint
Flinstones
Foreman
House
Lena
Peppers
Mean(all)

ReconNet
15.61dB
18.93dB
19.08dB
18.82dB
17.51dB
15.01dB
14.14dB
22.03dB
20.30dB
18.51dB
17.39dB
17.94dB
21.49dB
23.36dB
22.17dB
24.56dB
21.54dB
20.99dB
19.04dB
29.02dB
26.74dB
24.48dB
22.72dB
23.28dB
24.95dB
26.66dB
23.58dB
27.83dB
23.48dB
26.15dB
22.74dB
32.08dB
29.96dB
27.47dB
25.74dB
26.42dB

DR2 -Net
15.33dB
18.01dB
18.65dB
18.67dB
17.08dB
14.73dB
14.01dB
20.59dB
19.61dB
17.97dB
16.90dB
17.44dB
23.10dB
23.94dB
22.69dB
25.58dB
22.46dB
22.03dB
21.09dB
29.20dB
27.53dB
25.39dB
24.32dB
24.32dB
27.95dB
28.73dB
25.77dB
30.09dB
25.62dB
27.65dB
26.19dB
33.53dB
31.83dB
29.42dB
28.49dB
28.66dB

Adp-Rec
17.70dB
21.67dB
21.36dB
21.09dB
19.74dB
16.22dB
16.12dB
25.53dB
22.93dB
21.49dB
19.75dB
20.33dB
26.65dB
27.59dB
24.28dB
28.80dB
24.97dB
26.55dB
23.83dB
33.51dB
31.43dB
28.50dB
26.67dB
27.53dB
29.25dB
30.51dB
27.40dB
32.47dB
27.11dB
32.31dB
27.94dB
36.18dB
34.38dB
31.63dB
29.65dB
30.80dB

Fully-Conv
17.98dB
21.80dB
21.61dB
21.73dB
19.88dB
16.24dB
16.55dB
25.18dB
22.93dB
21.77dB
20.80dB
20.59dB
25.20dB
26.82dB
24.39dB
28.52dB
24.58dB
26.92dB
23.08dB
31.96dB
30.81dB
27.76dB
26.69dB
26.98dB
28.47dB
29.90dB
27.11dB
31.75dB
26.73dB
30.92dB
27.02dB
35.08dB
33.63dB
30.65dB
29.71dB
30.09dB

Proposed
18.46dB
22.49dB
22.06dB
22.3dB
20.63dB
16.33dB
16.92dB
27.26dB
23.67dB
22.51dB
21.38dB
21.27dB
27.61dB
27.92dB
24.28dB
29.48dB
25.62dB
27.36dB
24.98dB
34.00dB
32.36dB
28.97dB
28.72dB
28.30dB
32.63dB
32.13dB
28.59dB
33.88dB
28.99dB
32.91dB
30.26dB
38.10dB
36.22dB
33.00dB
32.90dB
32.69dB

5. Conclusion
This paper proposes a novel CNN-based deep neural network for highquality compressive sensing image reconstruction. The network uses a fully-

11

convolutional architecture, which removes the block effect caused by block-wise
methods. For a further improvement, we add Resblock after the deconvolutional
layer, making the network learn the residual information between low and high
resolution images. With this enhancement, the network shows best performance
in reconstruction task compared with other methods. In future work, we are
going to apply perceptual loss into the network for better reconstruction result.
And semantics-oriented reconstruction will be also considered.

References
[1] R. G. Baraniuk, V. Cevher, M. F. Duarte, C. Hegde, Model-based compressive sensing, IEEE Trans. Inform. Theory 56 (4) (2010) 1982–2001.
[2] R. G. Baraniuk, More is less: Signal processing and the data deluge, Science
331 (6018) (2011) 717–719.
[3] E. Candes, J. Romberg, Sparsity and incoherence in compressive sampling,
Inverse Probl. 23 (3) (2007) 969.
[4] J. Haupt, W. U. Bajwa, G. Raz, R. Nowak, Toeplitz compressed sensing matrices with applications to sparse channel estimation, IEEE Trans.
Inform. Theory 56 (11) (2010) 5862–5875.
[5] D. Needell, J. A. Tropp, Cosamp: iterative signal recovery from incomplete
and inaccurate samples, Commun. ACM 53 (12) (2010) 93–100.
[6] S. D. Babacan, R. Molina, A. K. Katsaggelos, Bayesian compressive sensing
using laplace priors, IEEE Trans. Image Processing 19 (1) (2010) 53–63.
[7] E. J. Candes, T. Tao, Decoding by linear programming, IEEE Trans. Inform. Theory 51 (12) (2005) 4203–4215.
[8] M. A. Figueiredo, R. D. Nowak, S. J. Wright, Gradient projection for sparse
reconstruction: Application to compressed sensing and other inverse problems, IEEE J. Sel. Top. Sign. Proces. 1 (4) (2007) 586–597.

12

[9] D. L. Donoho, A. Maleki, A. Montanari, Message-passing algorithms for
compressed sensing, P. Natl. A. Sci. 106 (45) (2009) 18914–18919.
[10] I. Daubechies, M. Defrise, C. De Mol, An iterative thresholding algorithm
for linear inverse problems with a sparsity constraint, Commun. Pur. Appl.
Math. 57 (11) (2004) 1413–1457.
[11] A. Mousavi, A. B. Patel, R. G. Baraniuk, A deep learning approach to
structured signal recovery, in: Annual Allerton Conference on Communication, Control, and Computing, IEEE, 2015, pp. 1336–1343.
[12] K. Kulkarni, S. Lohit, P. Turaga, R. Kerviche, A. Ashok, Reconnet: Noniterative reconstruction of images from compressively sensed measurements,
in: Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2016, pp. 449–458.
[13] A. Mousavi, R. G. Baraniuk, Learning to invert: Signal recovery via deep
convolutional networks, in: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2017.
[14] A. Mousavi, G. Dasarathy, R. G. Baraniuk, Deepcodec: Adaptive sensing
and recovery via deep convolutional neural networks, in: arXiv preprint
arXiv:1707.03386, 2017.
[15] X. Xie, Y. Wang, G. Shi, C. Wang, J. Du, X. Han, Adaptive measurement
network for cs image reconstruction, in: China Conference on Computer
Vision, 2017.
[16] A. Stevens, N. D. Browning, Less is more: Bigger data from compressive
measurements, Microscopy and Microanalysis 23 (S1) (2017) 166–167.
[17] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and
pattern recognition, 2016, pp. 770–778.

13

[18] H. Yao, F. Dai, D. Zhang, Y. Ma, S. Zhang, Y. Zhang, DR2 -net: Deep
residual reconstruction network for image compressive sensing, in: arXiv
preprint arXiv:1702.05743, 2017.
[19] S. Lohit, K. Kulkarni, R. Kerviche, P. Turaga, A. Ashok, Convolutional
neural networks for non-iterative reconstruction of compressively sensed
images, in: arXiv preprint arXiv:1708.04669, 2017.
[20] L. Xu, J. S. Ren, C. Liu, J. Jia, Deep convolutional neural network for image
deconvolution, in: Advances in Neural Information Processing Systems,
2014, pp. 1790–1798.
[21] B. Lim, S. Son, H. Kim, S. Nah, K. M. Lee, Enhanced deep residual networks for single image super-resolution, in: IEEE Conference on Computer
Vision and Pattern Recognition Workshops, 2017.
[22] K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask r-cnn, in: IEEE International Conference on Computer Vision, 2017.
[23] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,
S. Guadarrama, T. Darrell, Caffe: Convolutional architecture for fast feature embedding, in: Proceedings of the 22nd ACM international conference
on Multimedia, 2014, pp. 675–678.
[24] R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, L. Zhang, B. Lim,
S. Son, H. Kim, S. Nah, K. M. Lee, et al., Ntire 2017 challenge on single image super-resolution: Methods and results, in: IEEE Conference on
Computer Vision and Pattern Recognition Workshops, 2017, pp. 1110–
1121.

14

