Perceptual Compressive Sensing
Xuemei Xie*, Jiang Du, Guangming Shi, Chenye Wang, Xun Xu
School of Artificial Intelligence
Xidian University

arXiv:1802.00176v1 [] 1 Feb 2018

xmxie@mail.xidian.edu.cn

Abstract

ture information of the image. Instead of using the per-pixel
loss, we reconstruct images from the feature domain to capture the perceptual information. This makes it possible for
the network to learn the semantic information from natural
images. Experiments show that the CS image can be reconstructed in the feature of interests with the guiding of
semantics-oriented learning.
The organization of this paper is as follows. Related
works are introduced in Section 2. In Section 3, we present
the main technical design of the proposed fully convolutional neural network with perceptual loss of full image.
We present the experimental results in Section 4. Finally,
we draw the conclusion and further work in Section 5.

This paper proposes perceptual compressive sensing.
The network is composed of a fully convolutional measurement and reconstruction network. For the following contributions, the proposed framework is a breakthrough work.
Firstly, the fully-convolutional network measures the full
image which preserves structure information of the image
and removes the block effect. Secondly, with the employment of perceptual loss, we no longer concentrate on the
Euclidean distance of reconstruction. Instead, we just care
about the perceptual invariance, which makes the reconstructed image obtain much semantic information. We call
it semantics-oriented reconstruction. Experimental results
show that the proposed framework outperforms the existing
state-of-the-art methods.

2. Related work
CNN-based CS With the development of deep learning
technology, convolutional neural network is widely applied to data-driven compressive sensing problem in recent
years. ReconNet [6] firstly proposes the approach of using
a deep neural network for compressive sensing reconstruction. DR2 -Net [15] gives an improvement of an alternative
selection of the reconstruction network, where the number
of residual blocks is adjustable. Baraniuk et al [11] proposes DeepCodec, where the measurement network is adaptive, which breaks the limitation of using Gaussian random
measurement network. Adp-Rec [14] combines the advantage of adaptive measurement and deep learning based reconstruction, with jointly training of the whole framework,
which is a breakthrough work. In terms of loss function,
most of the related works focus on pixel-wise loss, while
new ReconNet also apply adversarial loss to restrain blurry
effects to some degree.

1. Introduction
Compressive sensing (CS) theory provides an idea to
reconstruct signals with high probability, although signals
are subsampled under Nyquist rates, with the assumption
that they are sparse in a certain domain. The CNN-based
methods [6, 10, 11, 14, 8, 15] learning a nonlinear mapping
function that fast reconstruct the CS image instead of the
optimization process by conventional methods using iterative algorithms [1, 2]. Newly proposed adaptive measurement methods [11, 14, 8] are used to acquire the measurement signals adaptively according to their natural characters. This makes retaining more information and achieving
better reconstruction results. However, all these improvements in CS use the per-pixel loss while measuring the similarity of the reconstructed images and the ground-truth images, which ignores the perceptual information.
In this paper, we propose a perceptual compressive sensing network. The framework of the proposed network is
shown in Fig. 1. By introducing a convolutional layer, the
full image is measured directly. Different from the blockwise operation, adopted in all the methods existing in CS
reconstruction task, the proposed method retains the struc-

Perceptual loss function Many classic image reconstruction tasks employ the per-pixel loss to measure the difference between the output and the ground-truth images.
These per-pixel loss are sensitive to the images offset since
they do not capture the perceptual differences between the
images to be measured. Recent works have shown that considering the perceptual differences extracted from a classi1

Figure 1. The framework of perceptual compressive sensing

one.

fication network can reconstruct higher quality images than
per-pixel differences. Mahendran et al. [9] use a feature reconstruction loss to invert features from different layers of
convolutional network, in order to understand the internal
representation in CNN. In the work of Johnson et al. [5],
they apply perceptual loss with VGG-16 network [12] to
style transfer and super-resolution tasks. It can reconstruct
the image with high-level semantic information. Thus, their
task of style transfer can manage to totally modify the style
(especially the detailed texture) without dropping the objects on the content image.

Fully convolutional architecture In the proposed framework, an adaptive convolution measurement is employed
to measure the full image. The feature map can preserve
the correspondence among pixels (instead of reshaping into
column vector). Thus, the block effect is removed due to
the overlapped convolutional measurement. As a symmetry, a deconvolution layer is right after the measurement to
recover the dimension of the feature map.
Residual reconstruction network Although the convolution and deconvolution layer can reconstruct the image,
for better accuracy, we enhance the proposed framework
with a residual network. Since batch normalization would
get rid of range flexibility from networks [7], we remove the
batch normalization layer in the residual network. In detail,
we just add one residual block and the it works well, as is
shown in Fig. 1. With the application of residual network,
the performance of the proposed framework is greatly enhanced.

3. Perceptual compressive sensing network
In this section, we mainly introduce the architecture of
proposed fully convolutional compressive sensing image reconstruction network with perceptual loss for full image.
The framework is shown in Fig. 1. It consists of two parts:
convolutional measurement network and perceptual reconstruction network.

3.1. Convolutional measurement network
Existing methods of compressive sensing always process
the image by dividing it into blocks, where the block is reshaped into column vector [3]. However, this will break the
structure information of the image. For an improvement,
we propose a new architecture.
In the compressive sensing network, the first convolution
layer is the measurement network, which is just a simple
production operation. Then, as a parallel structure, a deconvolution layer is here to recover the measurements to a lowresolution image of the original size. After that, a residual
network composed of three convolution layers is used here
to generate a high-resolution image from the low-resolution

3.2. Perceptual reconstruction network
Perceptual loss In prior works of CNN-based compressive sensing, the loss function is usually defined with pixelwise loss:
2
`pixel (ŷ, y) = kŷ − yk2
(1)
However, when it comes to the low measurement rate,
the reconstructed image is blurry. This is because pixelwise loss will force the image to have the minimized average Euclidean distance between the reconstruction and the
label, where more smooth texture of the image has better
2

(a) Original

(b) MSE

(c) Proposed (VGG 2 2)

(d) Proposed (VGG 3 4)

Figure 2. The framework of perceptual compressive sensing.

opportunity. Actually, when applying compressive sensing at an extremely low measurement rate, we do not care
much about the detailed texture of it. Correspondingly, we
will emphasize the importance of the structural information. Considering the current popular classification network
works by exerting the features in an image, we transfer this
advantage to our proposed method. Thus, we apply the perceptual loss. A typical kind of perceptual loss is the feature
loss. The feature loss is defined with the Euclidean distance
of the pre-trained VGG-19 network [12] between the feature maps of Res-Net output and label:
2

`φ,j
f eat (ŷ, y) = kφj (ŷ) − φj (y)k2

test image. Here we take the image dog as an example. In
Fig. 3, (a) is the original image. (b), (c) and (d) are respectively the reconstruction result of compressive sensing with
MSE loss, VGG 2 2 loss, and VGG 3 4 loss. In the original image, there is an eyeball in the dogs eye as is shown
in the bounding box. Then, we apply measurement rate at
1%. With MSE loss, the reconstruction result has the highest PSNR value, but in terms of visual effects, it is obviously
blurry. However, with perceptual loss, we surprisingly find
that the structure information is emphasized while detailed
texture is ignored.
With the application of VGG-19 as the perceptual loss,
the proposed network can reconstruct the image with the
pruning the network.

(2)

where φj (y) denotes the feature map of the j − th layer
with the input image y.In practical, we usually use VGG
2 2 or VGG 3 4 (actually pooling 2 or pooling 3) to define
the loss function. We do not apply perceptual loss of very
high level because in terms of compressive sensing, higher
level drops too much information that it is hardly possible
to inverse, even if pre-trained.
When different scale of feature map in VGG-19 is set
as the perceptual loss, we can compare the results on the

4. Experiments
In this section, we conduct the experiments on both MSE
loss and perceptual loss with fully convolutional neural network for compressive sensing problem. In addition, the proposed framework is applied to hardware.
3

(a) Original

(b) MSE

(c) Proposed (VGG 2 2)

(d) Proposed (VGG 3 4)

Figure 3. Experimental results of image ‘buildings’ with different methods with MR=0.04

Software simulation We use the caffe framework [4]
for training the network and MATLAB for testing. Our
computer is equipped with Intel Core i7-6700K CPU with
frequency of 4.0GHz, 4 NVIDIA GeForce GTX Titan V
GPUs, 128 GB RAM, and the framework runs on the
Ubuntu 16.04 operating system. The training dataset consists of 800 pieces of 256256 size images down sampled
and cut from 800 images in DIV2K dataset [13].
The experimental results of the proposed method compared to existing compressive sensing network are shown
in Fig. 3.
From Fig. 3, we can see that when the MR is set to 0.04,
compressive sensing with MSE loss gives a blurry reconstruction result, where the structure of the image can hardly
be recognized. For example, the windows of the building
are totally blurry and it is hard to count it. With the application of perceptual loss, more structural information can
be reconstructed at the same MR. Higher level of VGG loss
implies higher level of perception. As is shown in Fig. 3(d),
the broader of the windows is more clear and we can easily count them. Moreover, the visual effect is more pleasant
when it comes to higher level of perceptual loss. The hardware framework is shown in Fig. 6. Similar to prior work
ReconNet, we use an imaging arm to capture the light to the
measuring device. Instead of discrete micro-mirror device

(DMD), which is a binary instrument, we apply spatial light
modulators (SLM) to measure and the convolutional kernel is applied onboard. We use time-multiplexed method
to change the kernel. Considering the value of weights in
convolution layer can be either positive and negative, we
measure one kernel twice: Firstly the negative value (with
a inverter for symbol and a delay unit for synchronization)
and secondly the positive ones. Then, an amplifier is used
to sum up the values as the measurement. After that, FPGA
is used to calculate the forward propagation of the network.

5. Conclusion
In this paper, we propose a state-of-the-art framework of
perceptual compressive sensing. We apply perceptual loss
on full image for compressive sensing with pruned loss network. This is the first time of trial in this field, which may
open a door towards alternative to compressive sensing with
semantic reconstruction. In addition, an improved hardware
framework is proposed for better accuracy. In the future,
we will apply this technology to real-time perceptual video
compressive sensing and semantic communication.
4

(a) Original

(b) ReconNet [6] (18.93dB)

(a) Original

(b) ReconNet (18.19dB)

(c) Adp-Rec [14] (21.67dB)

(d) Proposed MSE(22.49dB)

(c) DR2 [15] (18.93dB)

(d) Proposed MSE(22.52dB)

(e)
Proposed
(19.38dB)

VGG2 2

(f)
Proposed
(18.07dB)

VGG3 4

(e)
Proposed
(18.25dB)

Figure 4. The reconstruction results of boats at measurement rate
1%.

VGG2 2

(f)
Proposed
(16.35dB)

VGG3 4

Figure 5. The reconstruction results of Monarch at measurement
rate 4%.

References
[1] E. J. Candes and T. Tao. Decoding by linear programming.
IEEE Trans. Inform. Theory, 51(12):4203–4215, 2005.
[2] M. A. Figueiredo, R. D. Nowak, and S. J. Wright. Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems. IEEE J. Sel. Top.
Sign. Proces., 1(4):586–597, 2007.
[3] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask rcnn. In IEEE International Conference on Computer Vision,
2017.
[4] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional
architecture for fast feature embedding. In ACM International Conference on Multimedia, pages 675–678, 2014.
[5] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for
real-time style transfer and super-resolution. In European
Conference on Computer Vision, pages 694–711. Springer,
2016.

Figure 6. Hardware application

[6] K. Kulkarni, S. Lohit, P. Turaga, R. Kerviche, and A. Ashok.
Reconnet: Non-iterative reconstruction of images from compressively sensed measurements. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition,
pages 449–458, 2016.
[7] B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee. Enhanced
deep residual networks for single image super-resolution. In
IEEE Conference on Computer Vision and Pattern Recogni-

5

tion Workshops, 2017.
[8] S. Lohit, K. Kulkarni, R. Kerviche, P. Turaga, and A. Ashok.
Convolutional neural networks for non-iterative reconstruction of compressively sensed images. In arXiv preprint
arXiv:1708.04669, 2017.
[9] A. Mahendran and A. Vedaldi. Understanding deep image
representations by inverting them. In Proceedings of the
IEEE conference on computer vision and pattern recognition, pages 5188–5196, 2015.
[10] A. Mousavi and R. G. Baraniuk. Learning to invert: Signal
recovery via deep convolutional networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 2272–2276, 2017.
[11] A. Mousavi, G. Dasarathy, and R. G. Baraniuk. Deepcodec:
Adaptive sensing and recovery via deep convolutional neural
networks. In arXiv preprint arXiv:1707.03386, 2017.
[12] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556, 2014.
[13] R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang,
L. Zhang, B. Lim, S. Son, H. Kim, S. Nah, K. M. Lee,
et al. Ntire 2017 challenge on single image super-resolution:
Methods and results. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1110–1121,
2017.
[14] X. Xie, Y. Wang, G. Shi, C. Wang, J. Du, and X. Han. Adaptive measurement network for cs image reconstruction. In
China Conference on Computer Vision, 2017.
[15] H. Yao, F. Dai, D. Zhang, Y. Ma, S. Zhang, and Y. Zhang.
DR2 -net: Deep residual reconstruction network for image
compressive sensing. In arXiv preprint arXiv:1702.05743,
2017.

6

