Recurrent Generative Residual Networks for Proximal Learning and Automated
Compressive Image Recovery

arXiv:1711.10046v1 [cs.AI] 27 Nov 2017

Morteza Mardani1 , Hatef Monajemi2 , Vardan Papyan2 , Shreyas Vasanawala3 ,
David Donoho2 , and John Pauly1
Electrical Engineering1 , Statistics2 , and Radiology3 Depts., Stanford University
morteza,monajemi,papyan,vasanawala,donoho,pauly@stanford.edu

Abstract

ford only a low overhead for inference and training; e.g.,
imagine MRI visualization for neurosurgery [1], or, realtime superresolution that may need re-training on a cell
phone [24]; c2) the need for recovering plausible images
that are consistent with the physical model; this is particularly important for medical diagnosis, which is sensitive to
hallucination.
Conventional compressed sensing (CS) relies on sparse
coding of images in a proper transform domain via a
universal `1 -regularization; see e.g., [8, 9, 19]. To automate the time-intensive iterative soft-thresholding algorithm (ISTA) used for sparse coding, [12] proposed learned
ISTA (LISTA). Relying on soft-thresholding it trains a simple (single-layer and fully-connected) recurrent network to
map measurement to a sparse code. Deep generative networks have also proven tremendously powerful in modeling
prior distribution for natural images [11, 15, 16, 21, 31, 32].
In particular, residual networks (ResNets) are commonly
used due to their stable behavior [14] along with pixel-wise
and perceptual costs induced by generative adversarial networks (GANs) [11]. There is a handful of recent attempts
to integrate the priors offered by generative nets for inverting linear inverse tasks dealing with local image restoration such as super-resolution [15, 16], inpainting [31]; and
more global tasks such as biomedical image reconstruction [7,17,20,22,25,27,29,33]. For instance, deep ResNets
are adopted with GANs to superresolve natural images with
state-of-the art perceptual quality [15, 16, 26].
For biomedical images one typically knows a small fraction of projections onto a certain transform domain (e.g.,
Fourier, or, Radon) based on physics of the scanner. Variations of deep CNNs are trained to map out aliased MR
(AutoMap [33]) or low-dose CT (RED-CNN [7]) images to
the gold-standard ones retrieved by iterative CS. They offer
rapid reconstruction at the expense of high training overhead. There is however no systematic mechanism to assure
fidelity to the underlying physical model, which can possibly hallucinate images. Another line of work pertains to developing effective priors to incorporate in an iterative algo-

Recovering images from undersampled linear measurements typically leads to an ill-posed linear inverse problem, that asks for proper statistical priors. Building effective priors is however challenged by the low train and test
overhead dictated by real-time tasks; and the need for retrieving visually “plausible” and physically “feasible” images with minimal hallucination. To cope with these challenges, we design a cascaded network architecture that unrolls the proximal gradient iterations by permeating benefits from generative residual networks (ResNet) to modeling
the proximal operator. A mixture of pixel-wise and perceptual costs is then deployed to train proximals. The overall architecture resembles back-and-forth projection onto
the intersection of feasible and plausible images. Extensive
computational experiments are examined for a global task
of reconstructing MR images of pediatric patients, and a
more local task of superresolving CelebA faces, that are insightful to design efficient architectures. Our observations
indicate that for MRI reconstruction, a recurrent ResNet
with a single residual block effectively learns the proximal.
This simple architecture appears to significantly outperform
the alternative deep ResNet architecture by 2dB SNR, and
the conventional compressed-sensing MRI by 4dB SNR with
100× faster inference. For image superresolution, our preliminary results indicate that modeling the denoising proximal demands deep ResNets.

1. Introduction
Linear inverse problems widely appear in image restoration tasks in applications ranging from super-resolving natural images to reconstructing biomedical images. In such
applications, one oftentimes encounters a seriously illposed recovery task, which necessitates regularization with
proper statistical priors. This is however impeded by the
following challenges: c1) real-time and interactive tasks af1

rithm which can outperform the conventional sparsity priors
for CS; see e.g., [2, 3, 5, 13, 27]. For instance, [5, 13] uses
the low-dimensional code offered by a pre-trained generative decoder to achieve higher SNR than CS. These schemes
attain a reasonably high SNR, but need several iterations for
convergence that hinders real-time imaging.
Toward rapid, feasible, and plausible image recovery for
ill-posed linear inverse tasks, this paper proposes a novel
approach to automate imaging. Inspired by proximal gradient iterations, a recurrent ResNet architecture is designed
to learn the proximal(s) from the data. Image prior information is learned through the proximal, that is modeled as
a generator (G) network, consisting a few residual blocks
(RB), trained with mixture of pixel-wise and perceptual
costs via GANs to recover plausible images. The overall
architecture implements multiple back-and-forth (approximate) projections onto the subspace of data-consistent images - dictated by the physical model - and the manifold
of plausible images. The number of projections and similarly the size of each G network is desired to be small to reduce the training and inference overhead for real-time and
interactive image recovery tasks. To study this effect, we
perform experiments for reconstructing pediatric MR images, and super-resolving natural images from the CelebA
face dataset. The former is a global task with subsampled
measurements in the frequency domain, so-termed k-space,
while the latter is a rather local task, with a pixelated lowresolution image.
Our observations indicate that for MRI reconstruction,
one better repeat a small ResNet (with a single RB) multiple times, instead of training a deep network, as commonly
used e.g., in [17, 25, 29, 33]. The recurrent architecture not
only improves upon the deep schemes by about 2dB SNR,
but also incurs much less training overhead. This simple
architecture also significantly outperforms the conventional
CS-Wavelet/CS-TV schemes; 4dB SNR gain in 100 times
shorter time. For the local 4x superresolution task however
it turns out that one needs to train a deep ResNets to learn
the denoising proximals, and the preliminary results do not
show any major advantage in using recurrent schemes. Before closing this section, it is worth to re-iterate the main
contributions of this paper as follows:
• Learning proximals using ResNet generators trained
with pixel-wise and GAN-based perceptual costs
• Extensive experiments for MRI reconstruction and
face superresolution gives insight to design proper network architectures for various recovery tasks
The rest of this paper is organized as follows. Section 2
states the problem. Proximal learning based on recurrent
GANs is discussed in Section 3. Evaluations for pediatric
MR image reconstruction, and natural image superresolu-

tion are reported in Section 4, while the conclusions are
drawn in Section 5.

2. Preliminaries and problem statement
Consider an ill-posed linear system y = Φx + v with
Φ ∈ CM ×N where M  N , and v captures the noise and
unmodeled dynamics. Suppose the unknown and (complexvalued) image x lies in a low-dimensional manifold, say M.
No information is known about the manifold besides the
training samples X := {xk }K
k=1 drawn from it, and the corresponding (possibly) noisy observations Y := {yk }K
k=1 .
Given a new undersampled observation y, the goal is to
quickly recover a plausible x.
The stated problem covers a wide range of image restoration and reconstruction tasks. For instance, in medical image reconstruction Φ describes a projection driven
by physics of the acquisition system (e.g., Fourier transform for MRI scanner). For image superresolution it
is the downsampling operator that averages out nonoverlapping image regions to arrive at a low-resolution one.
Given the image prior distribution, one typically forms a
maximum-likelihood estimator formulated as a regularized
least-squares (LS) program
(P1)

min
x

y − Φx

2

+ ψ(x; Θ)

(1)

with the regularizer ψ(·) parameterized by Θ that incorporates the image prior.
In order to solve (P1) one can adopt a variation of proximal gradient algorithm [23] with a proximal operator Pψ {·}
that is obtained based on ψ(·) [23]. Starting from x[0] = 0,
and adopting a small step size α the overall iterative procedure is expressed as
n
o
x[k + 1] = Pψ x[k] + αΦH (y − Φx[k])
o
n
(2)
= Pψ αΦH y + (I − αΦH Φ) x[k]
{z
}
|
:=PN

For convex proximals the fixed point of 2 coincides with
the global optimum for (P1) [23]. For some simple prior
distributions, the proximal operation is convex and tractable
in closed-form. One popular example of such a proximal
pertains to `1 -norm regularization for sparse coding, where
the proximal operator gives rise to soft-thresholding and
shrinkage in a certain domain such as Wavelet, or, totalvariation (TV). The associated iterations goes by ISTA,
which are then improved to FISTA iterations with accelerated convergence [4].
As argued earlier FISTA is a universal and thus naive
regularization that does not take into account the image
complications and perceptual quality. In addition, for moderate and high resolution images it demands many iterations

for convergence that can seriously imped real-time recovery. The next sections aim to fix these caveats by learning
proximals from historical images using generative neural
networks.

3. Proximal learning
Motivated by the proximal gradient iterations in (2), to
design efficient network architectures that automatically invert linear tasks, we need to first address the following important questions:
• How can one ensure the network does not hallucinate
images, and retrieves plausible images that are physically feasible?
• How can one ensure rapid inference and affordable
training for real-time and interactive image recovery
tasks?

3.1. Recurrent network architecture
The recursion in (2) can be envisioned as a feedback loop
in Fig. 1, which takes an initial image estimate x̃ that is subsequently projected onto the manifold M through the proximal operator to return x̂. The proximal operator is modeled
via a generative neural networks as will be elaborated in the
next section. Projection onto M is supposed to remove artifacts to some extent, and result in a more visually appealing
image. To close the loop, x̂ then passes through the filter
PN := I − αΦH Φ and is added up to the input x̃. The
feedback filter PN resembles projection onto the nullspace
of the measurement operator. Notice, PN only needs tunning the step size α that can be easily learned from the data.
Implementing the feedback loop however demands (possibly) infinitely many iterations.
To bypass this hurdle, inspired by recurrent neural networks (RNNs) we unroll the loop and repeat multiple, say
K, copies of the proximal network as depicted in Fig. 1
(bottom). Each proximal network is accompanied with
an (approximate) data consistency projection that refines
x̌ to be consistent with the observations by simply moving along the descent direction of the data fidelity cost,
namely ky − Φxk2 . Assuming exact data consistency projection the unrolled network learns the projection onto the
intersection of physically feasible and visually plausible images. In general one can consider the cascaded architecture
in Fig. 1 with independent weights {Θk }K
k=1 per copies,
but we are more interested in sharing the weights, namely
Θ1 = . . . = ΘK , which needs less training variables, and
the back-propagation can easily accommodate gradient calculations [10].
In essence, (multiple) back-and-forth projections can ensure data fidelity to a good extent. This is in contrast with
the existing deep architectures for automated medical image

reconstruction (e.g., [7, 20, 33]) with no consideration for
data fidelity, which may hallucinate images and mislead the
diagnosis. It is also worth mentioning that the AmortisedMAP based deep-GAN scheme in [26] uses an affine projection layer that improves GAN’s stability and suerresolution quality. However, one naturally need multiple projections to assure data consistency. The number of copies
however cannot be large, or, alternatively the proximal networks need to be small, for real-time inference tasks.

3.2. Mixture of pixel-wise and perceptual costs
To learn proximals as projections onto manifold of visually plausible images we adopt GANs [11]. Conventional
generative models such as variational auto-encoders [16]
rely on pixel-wise costs that offer high pick signal-to-noise
ratios but often produce overly-smooth images with poor
perceptual quality. GANs however train a perceptual loss
from the training data. Standard GANs consist of a tandem structure of generator (G) and discriminator (D) networks [11].
Training GANs amounts to playing a game with conflicting objectives between the adversary G and the discriminator. D network aims to score one the training groundtruth images drawn from the data distribution, and zero the
(fake) outputs of G. Apparently, D cannot perfectly separate real and fake images as G tries to generate fake images
that fools G. Various strategies have been devised to reach
the game’s equilibrium. They mostly differ in evaluating
the loss incurred by G and D [11], [21]. The conventional
GAN [11] uses a sigmoid cross-entropy for D’s loss, which
suffers from vanishing gradients. It leads to unstable training that causes mode collapse. In addition, for the generated
images classified confidently as real (with a large decision
variable), no cost is incurred. Hence, it tends to pull samples away from the decision boundary, that introduces nonrealistic images [21]. This particularly can hallucinate medical images, and as a result mislead medical diagnosis. To
alleviate this issue, we adopt least-square GAN (LSGAN)
that penalizes the classification mistake with a LS cost that
pulls the generated samples towards the decision boundary.
One issue with GAN however is that it may overemphasize high frequencies at the expense of deteriorating the image main structure. To avoid this issue, along
with LSGAN perceptual loss we use a pixel-wise `1 /`2 cost,
which perform well in maintaining the structure, and discarding the low-intensity noise [32]. Training G with the
mixture cost is thus expected to reveal fine texture details
while discarding noise. Let us collect all the network parameters in Θ := (Θg , Θd ). The overall procedure then
aims to jointly minimize the discriminator cost
h
2 i
h
2 i
(P1.1) min Ex 1 − D(x; Θd )
+ Ey D(x̂; Θd )
Θ

and the generator cost

Figure 1. Recurrent GAN. (Top) the closed-loop circuit diagram, and (right) unrolled computational graph with K copies.
K
hX

i
h
2 i with 3 × 3 kernels and a fixed number of 128 feature maps,
+λEy 1 − D x̂; Θd
respectively, that are followed by batch normalization (BN)
Θ
k=1
and ReLU activation. It is then followed by three simple
h
i
convolutional layers with 1 × 1 kernels, where the first two
+ ηEx,y x − x̂ 1,2
(3)
layers undergo ReLU activation and the last layer has sigmoid activation to return the output; see Fig. 2. Notice that
where kxk1,2 := γkxk1 + (1 − γ)kxk2 for some 0 ≤ γ ≤
for all generators {G(Θk )}K
1. The LS data fidelity term in (P1.2) is a soft version of
k=1 a similar ResNet architecture is used.
the affine projection in the network architecture of Fig. 1.
The D network is composed of eight convolutional layParameter λ is also tuned based on the measurement noise
ers. In all the layers except the last one, the convolution is
level and the expected pixel-wise fidelity.
followed by BN and ReLU activation. No pooling is used.
4. Experiments
For the first four layers, number of feature maps is doubled
from 8 to 64, while at the same time convolution with stride
Performance of the novel recurrent GANCS scheme is
2 is used to reduce the image resolution. Kernel size 3×3 is
assessed in reconstructing pediatric MR images and superadopted for the first five layers, while the last two layers use
resolving natural images. The former introduces aliasing
kernel size 1 × 1. In the last layer, the convolution output
artifacts that globally impact the entire image pixels, while
is averaged out to form the decision variable for LS binary
in the latter the pixelation occurs locally. While the focus is
classification, where no soft-max is used.
mostly placed on MRI, preliminary results are also reported
Adam optimizer is used with the momentum parameter
for image super-resolution to shed some light on challenges
β = 0.9, mini-batch size Lb = 2, and learning rate µ =
associated with proximal learning. In particular, we aim to
10−5 . Training is performed with TensorFlow interface on
address the following intriguing questions:
a NVIDIA Titan X Pascal GPU with 12GB RAM.
Q1. What is the proper number of copies, and generator size
4.2. MRI reconstruction and artifact suppression
to learn the proximal?
Q2. What is the trade-off between PSNR/SSIM and inferPerformance of the novel recurrent scheme is assessed
ence/training complexity?
in removing aliasing artifacts from MR images. In essence,
Q3. How is the performance compared with the conventhe scanner acquires Fourier coefficients (k-space data) of
tional sparse coding?
the underlying image across various coils. A single-coil MR
Q4. How does the performance change if we train with inacquisition model is considered where for n-th patient the
dependent weights per copies, and what is the interpretation
acquired k-space data admits
for output of different copies?
(n)
(n)
yi,j = [F(Xn )]i,j + vi,j , (i, j) ∈ Ω
(4)
4.1. Residual generator networks and training

(P1.2) min Ey

y − Φx̌k

2

To address the above questions, for the generator networks we adopt a ResNet with a variable number of residual
blocks (RB). Each RB consists of two convolutional layers

Here, F refers to the 2D Fourier transform, and the set Ω
indexes the sampled Fourier coefficients. As it is conventionally performed with CS MRI, we select Ω based on a

Figure 2. Generator ResNet architecture with RBs, n and k refer to number of feature maps and filter size, respectively.

Figure 3. Discriminator multilayer CNN architecture with the input magnitude image, where n, k, and s refer to number of feature maps,
filter size, and stride size, respectively.

with voxel resolution 1.07 × 1.12 × 2.4 mm. The input
and output are complex-valued images of the same size and
each include two channels for real and imaginary components. The input image x̃ is simply generated using inverse
2D FT of the k-space data where the missing ones are filled
with zero (ZF), and is severely contaminated with artifacts.
In fact one may ponder whether the gold-standard, i.e., the
fully-sampled raw k-space data, is available. The answer
is affirmative. We average out the acquired time-resolved
contrast-enhanced k-space data over time to end up with
fully-sampled data. Then, we synthesize the undersampled
data by randomly selecting only 20% of the k-space pixels
based on the sampling mask in Fig. 4.
Figure 4. k-space sampling trajectory with 5-fold undersampling
based on variable density distribution with radial view ordering.

variable density sampling with radial view ordering that is
more likely to pick low frequency components from the center of k-space [19]. Only 20% of Fourier coefficients are
collected. The sampling mask is shown in Fig. 4.
Dataset. Contrast-enhanced T1-weighted abdominal image
volumes are acquired for 350 pediatric patients. Each 3D
volume includes 151 axial slices of size 200 × 100 pixels. 300 patients (45, 300 slices) are considered for training, and 50 patients (7, 550 slices) for test. All in vivo
scans were acquired on a 3T MRI scanner (GE MR750)

4.2.1

Performance for various number/size of copies

In order to assess the impact of network wiring on the image
recovery performance, the cascaded network is trained for
a variable number of ResNet copies with variable number
of RBs. 10k slices from the train dataset set are randomly
picked for training, and 1, 280 slices from the test dataset
for test.
Shared training weights. The copies are first assumed to
be identical with shared weights that results in a recurrent
network. Training is performed for various combinations of
GAN and pixel-wise costs. We report most of the results
with only `2 cost alone in (P1), i.e., γ = 0, η = 1, λ = 0,

where no D network is trained as it is easier to train and
evaluate the performance quantitatively for several network
architectures, which is an important purpose of this work.
We report more evaluations with GAN perceptual cost later.
Fig. 5 depicts the SNR and structural similarity index metric
(SSIM) [30] versus the number of copies, when each copy
comprises 1/2/5/10 RBs. It is observed that increasing
the number of copies significantly improves the SNR and
SSIM, but lead to a longer inference and training time. In
particular, using three copies instead of one achieves more
than 2dB SNR gain for 1 RB, and more than 3dB for 2 RBs.
It is also interestingly observed that when using a single
copy, adding more than 5 RBs to make a deeper network
does not improve anymore; look at the SNR=24.33 for 10
RBs, and SNR=24.15 for 5 RBs. Notice also that a single
RB is not also expressive enough to learn the MR image denoising proximal, and as a result repeating it several times,
the SNR does not seem to exceed 27dB. Using 2 RBs however turns out to learn the proximal, and perform as good as
using 5 RBs. Similar observations are made for SSIM.
Independent weights. We also consider a scenario where
one allows weights varying across different copies. A similar ResNet architecture is used for all copies, which multiplies the variable count for training by the number of copies.
As seen in Fig. 6, adopting a single RB per copy and repeating it for 15 − 20 copies seems to be a suitable choice that
achieves up to 27.4dB SNR. Apparently, a single RB and
10 copies performs as good as 4 − 5 RBs with 5 copies in
terms of SNR and SSIM. Comparing with the shared weight
scenario, for 10 copies with a single RB, using independent
weights improves the SNR by almost 1dB. Notice that when
each copy includes more than 5 RBs, our GPU resources become exhausted for more than 5 copies, and thus the rest of
points are not shown on the plot. Further evaluations with
more efficient implementation and stronger GPU resources
is deferred for our future research.
Training and inference time. Inference time for both
shared and independent weights is the same, and proportional to the number of copies. Feed-forwarding each image through a copy with one RB takes 4 msec when fully
using the GPU. The training variable count is also proportional to the number of copies when the weights are allowed
to change per different copies. It is hard to precisely evaluate the training and inference time under fair conditions as
it strongly depends on the implementation and the allocated
memory and processing power per run. As an estimate for
the inference time we average it out over a few runs on the
GPU as listed in Table 2. It is empirically observed that
with shared weights, e.g., 10 copies with 1 RB the training
converges in 2 − 3 hours, but a deep single copy ResNet
with 10 RBs takes around 10 − 12 hours to converge.

4.2.2

Comparison with sparse coding

To compare with the conventional CS schemes, CS-WV and
CS-TV are adopted and tunned for the best SNR performance using BART [28] that runs 300 iterations of FISTA
along with 100 iterations of conjugate gradient descent to
reach convergence. Quantitative results are listed under Table 2, where it is evident that the recurrent scheme with
shared weights significantly outperforms CS with more than
4dB SNR gain that leads to sharper images with finer texture
details as seen in Fig. 8. As a representative example Fig. 8
depicts the reconstructed abdominal slice of a test patient.
CS-WV retrieves a blurry image that misses out the sharp
details of the liver vessels. A deep ResNet with one copy
and 10 RBs captures a cleaner image, but still smoothens
out fine texture details such as vessels. However, when using 10 simple copies with a single RB, more details are seen
about the liver vessels, and the texture appears to be more
realistic. Similarly, using 5 copies each containing 2 RBs
retrieves finer details than 2 relatively large copies with 5
RBs.
This observation indicates that the proximal for denoising MR images is well represented by a small number 1 − 2
RBs. The important message however is that multiple backand-forth iterations are needed to recover a plausible MR
image that is physically feasible. considering the training and inference overhead as well as the quality of reconstructed image in Fig. 8, the architecture with 10 copies and
1 RB seems promising to implement in clinical scanners.
4.2.3

LSGAN for sharp MR images

We train the GAN scheme with the generator cost (P1.2)
that relies 90% on the pixel-wise `2 cost, and 10% on the
LSGAN cost. Recurrent LSGAN is trained with shared
weights per copies. To avoid mode collapse, we begin the
training with the pixel-wise `2 cost and gradually increase
the GAN loss weight to reach 10% after around 103 minibatches. This helps the GAN output to be consistent with
the images, and thus the generator initial distribution overlapping with the true image distribution. We found this
trick very useful in stably training GANs especially for the
proposed recurrent architecture. All the network architectures discussed next are seen to converge after a few dozens
epochs over the training data.
Fig.8 compares the retrieved images by various recurrent GAN architectures with the input ZF image as well
as the gold-standard one that is fully-sampled. Abdominal
slices shown for two representative axial slices including
liver and kidneys confirm again that RGANCS scheme with
10 copies and 1 RB performs the best in terms of perceptual
quality. Even though SNR and SSIM are not proper metrics to assess the perceptual quality, for the sake of completeness we report them in Table 2. This also corrobo-

Figure 5. Average SNR and SSIM versus the number of copies when the weights are shared among the copies.

Figure 6. Average SNR and SSIM versus the number of copies when the weights across copies are allowed to change independently.

rates even when using perceptual loss for training, recurrent scheme can significantly improve SNR/SSIM relative
to a single deep network (1 copy, 10 RBs) as commonly
adopted for image restoration tasks in the literature. The
RGANCS images are sharper than the CS-wavelet scheme,
even though CS achieves a higher SNR/SSIM. Choosing a
smaller weight λ, or, a larger η, RGANCS can even improve the SNR/SSIM as it was seen in Table 1 of the paper.
Further tunning of λ and η for the best performance needs
expert opinion of radiologists about the diagnostic quality
of the resulting images and is the subject of our ongoing
research.

4.3. Single image super-resolution
More evaluations are performed for super-resolving natural images. In essence, super-resolution can be seen as
a linear inverse task, where one has only access to a lowresolution image y = φ ∗ x + v obtained after downsampling with a convolution kernel φ. We adopt a 4 × 4 constant kernel with stride 4 that averages out the image pixel
intensities over 4×4 non-overlapping regions. Image superresolution is a challenging ill-posed problem, and has thus
been the subject of intensive research over the last decade;

see e.g., [6,15,24,26] and the references therein. [26] leverages deep convolutional GANs (DCGANs) accompanied
with an affine projection layer to find a better solution as
measured by SNR and SSIM. [16] also deploys a deep
ResNet (16 RBs with 64 feature maps) along with GAN
perceptual cost to retrieve photo-realistic images. Our goal
is not to create better looking images than the state-of-theart, but to study proximal learning for this application that
gives insights about possibly simpler network architectures
for real-time tasks, and can interpret the proximal behavior
in terms of revealing the details.
CelebA dataset. Adopting celebFaces Attributes Dataset
(CelebA) [18], for training and test we use 10k and 1, 280
images, respectively. Each ground-truth face image has
128 × 128 pixels that is down-sampled to a 32 × 32 lowresolution image.
Training. Our TensorFlow implementation uses a 2D conv
with stride 4 for downsampling, and transpose conv with
stride 4 for upsampling. Note, transpose convolution does
not perform deconvolution, and a single conv transpose can
be quite suboptimum. We approximate the deconvolution
pseudo-inverse with a few (5) gradient-descent iterations
with a small step size (0.1). The deconvolution then in-

Table 1. Performance trade-off for various architectures with shared and independent weights.

copies
10
5
2
1
CS-TV
CS-WV

RBs
1
2
5
10
n/a
n/a

inference time (sec)
0.04
0.10
0.12
0.0522
1.30
1.16

SNR (dB), independent
27.03
27.01
28.14
24.33
22.20
22.51

SSIM, independent
0.923
0.9258
0.944
0.8810
0.82
0.86

SNR (dB), shared
26.07
26.94
26.55
24.33
22.20
22.51

SSIM, shared
0.9117
0.9221
0.9194
0.8810
0.82
0.86

Figure 7. A representative axial abdominal slice for a test patient reconstructed by zero-filling (1st column); CS-WV (2nd column); and
RGANCS with 1 copy and 10 RBs (3rd column), 10 identical copies and 1 RBs (4th column), 2 identical copies and 5 RBs (5th column),
5 identical copies and 2 RBs (6th column); and the gold-standard (7th column).
Table 2. SNR and SSIM performance for various architectures
with shared weights when η = 0.9 and λ = 0.1.

copies
10
5
2
1

RBs
1
2
5
10

SNR (dB)
22.04
20.16
19.24
18.48

SSIM
0.835
0.794
0.7496
0.71

volves conv and transpose conv, and its gradient turns out
to below up abruptly and generates NaNs during gradient
backpropagation. We thus use gradient clipping for the G
network that fixes the issue. The network is fed with pixelated 128 × 128 images with three RGB channels obtained
by an approximate deconvolution. The same network architecture as for MRI is adopted.
4.3.1

Local versus global recovery tasks

For the superresolution task when sharing the weights no
interesting pattern is observed for ResNets of size 1 − 7
RBs, which indicates modeling the proximal needs larger

networks. For the scenario with varying weights Fig. 9 plots
PSNR and SSIM for various architectures. Using 5 copies
with 2 RBs seems to perform as good as a deep ResNet with
15 RBs adopted in [16]. However, increasing the number of
copies and RBs does not offer any clear advantages. This
is in contrast with MRI reconstruction where a recurrent
single RB could significantly outperform the deep architectures with up to 10 RBs. It appears that recurrent architecture sounds more useful for global recovery tasks where the
observation matrix entangles the image pixels. Perhaps by
going to k-space ResNet can better learn the proximals to
invert the map. This is deferred to future research.
4.3.2

Interpretation of generator outputs

Fig. 10 depicts the output of different generator copies when
training a network architecture with 4 independent copies,
each composed of 5 RBs. RGB images are shown in gray
scale. It is seen that different copies focus on features at
different levels of abstraction associated with different frequency components. The first block tries to retrieve major
(low-frequency) structural features at the expense of introducing a large amount of high-frequency noise, which is

Figure 8. Representative axial abdominal slices for a test patient reconstructed by zero-filling (1st column); and the recurrent GANCS
(RGANCS) with 1 copy and 10 RBs (2nd column), 10 identical copies and 1 RBs (3rd column), 2 identical copies and 5 RBs (4th column),
5 identical copies and 2 RBs (5th column); and the gold-standard (6th column). For RGANCS we used η = 0.9 and λ = 0.1.

Figure 9. Average SNR and SSIM for super-resolving CelebA face images for various number of copies when the weights are independent.

then washed away by the next copy. The third copy then
adds up high-frequency components to improve the sharpness, which introduces some noise that is again alleviated
by the fourth copy to retrieve the output. The overall process tend to alternate between sharpening and smoothing.

5. Conclusions and closing remarks
This paper caters a novel proximal learning framework
for automated recovery of images from compressed linear
measurements. Unrolling the proximal gradient iterations, a
recurrent/cascade architecture is devised that alternates between proximal projection and data fidelity. ResNets are
adopted to model the proximals, and a mixture of pixel-wise

and perceptual costs used for training. Experiments are examined to assess various network wirings in reconstructing
MR images of pediatric patients, and superresolving face
images. Our observations indicate that a recurrent small
ResNet can effectively learn the proximal, and significantly
improve the quality and complexity of recent deep architectures (single copy) and the conventional CS-MRI. Our preliminary results for single-image suprerresolution however
indicate that the recurrent architecture are not that effective
compared with the exiting deep schemes.
There are still unanswered questions that are the focus
of our current research. They pertain to running more experiments with perceptual costs with a subjective qualityassessment strategy; more extensive experiments for super-

Figure 10. From left to right: low-resolution input x̃, generator outputs x̌1 , x̌2 , x̌3 , x̂, respectively, for a representative test face image. All
images are displayed in gray scale. First row shows the output images with the corresponding frequency domain in the second row, when
trained using 4 independent ResNet copies with 5 RBs.

resoltuion with larger ResNet sizes and possibly training
in the k-space with more global measurements; and a fair
mechanism to compare the inference/training time.

[12]

References
[1] http://www.mriinterventions.com/clearpoint/clearpointoverview.html. 1
[2] J. Adler and O. Öktem. Learned primal-dual reconstruction.
arXiv preprint arXiv:1707.06474, 2017. 2
[3] J. Adler, A. Ringh, O. Öktem, and J. Karlsson. Learning
to solve inverse problems using Wasserstein loss. arXiv
preprint arXiv:1710.10898, 2017. 2
[4] A. Beck and M. Teboulle. A fast iterative shrinkagethresholding algorithm for linear inverse problems. SIAM
journal on imaging sciences, 2(1):183–202, 2009. 2
[5] A. Bora, A. Jalal, E. Price, and A. G. Dimakis. Compressed sensing using generative models. arXiv preprint
arXiv:1703.03208, 2017. 2
[6] J. Bruna, P. Sprechmann, and Y. LeCun. Super-resolution
with deep convolutional sufficient statistics. arXiv preprint
arXiv:1511.05666, 2015. 7
[7] H. Chen, Y. Zhang, M. K. Kalra, F. Lin, P. Liao, J. Zhou,
and G. Wang. Low-dose CT with a residual encoder-decoder
convolutional neural network (RED-CNN). arXiv preprint,
arXiv:1702.00288v2 [physics.med-ph], 2017. 1, 3
[8] D. L. Donoho. Compressed sensing. IEEE Transactions on
information theory, 52(4):1289–1306, 2006. 1
[9] J. M. Duarte-Carvajalino and G. Sapiro. Learning to sense
sparse signals: Simultaneous sensing matrix and sparsifying
dictionary optimization. IEEE Transactions on Image Processing, 18(7):1395–1408, 2009. 1
[10] I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning.
MIT Press, 2016. http://www.deeplearningbook.
org. 3
[11] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Gen-

[13]

[14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

[22]

erative adversarial networks. In Advances in Neural Information Processing Systems, Montreal, Canada, Dec. 2014. 1, 3
K. Gregor and Y. LeCun. Learning fast approximations of
sparse coding. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 399–406,
2010. 1
P. Hand and V. Voroninski. Global guarantees for enforcing deep generative priors by empirical risk. arXiv preprint
arXiv:1705.07576, 2017. 2
K. He, X. Zhang, S. Ren, and S. J. Identity mappings in
deep residual networks. arXiv preprint, arXiv:1603.05027v2
[cs.CV], Apr. 2016. 1
J. Johnson, A. Alahi, and F.-F. Li. Perceptual losses for
real-time style transfer and super-resolution. arXiv preprint,
arXiv:1603.08155v1 [cs.CV], Mar. 2016. 1, 7
C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham,
A. Acosta, A. Aitken, A. Tejani, J. Totz, W. Z., and S. W.
Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint, arXiv:1609.04802
[cs.CV], Sep. 2016. 1, 3, 7, 8
D. Lee, J. Yoo, and J. C. Ye. Compressed sensing and parallel
MRI using deep residual learning. In Proceedings of the 25st
Annual Meeting of ISMRM, Honolulu, HI, USA, 2017. 1, 2
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face
attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), Dec. 2015. 7
M. Lustig, D. Donoho, and J. M. Pauly. Sparse MRI: The application of compressed sensing for rapid mr imaging. Magnetic Resonance in Medicine, 58(6):1182–1195, 2007. 1, 5
A. Majumdar. Real-time dynamic MRI reconstruction
using stacked denoising autoencoder.
arXiv preprint,
arXiv:1503.06383 [cs.CV], Mar. 2015. 1, 3
X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley. Least-squares generative adversarial networks. arXiv
preprint, arXiv:1611.04076v3 [cs.CV], Apr. 2017. 1, 3
M. Mardani, E. Gong, J. Y. Cheng, S. Vasanawala, G. Zaharchuk, M. Alley, N. Thakur, S. Han, W. Dally, J. M. Pauly,
et al. Deep generative adversarial networks for compressed

[23]
[24]

[25]

[26]

[27]

[28]

[29]

[30]

[31]

[32]

[33]

sensing automates MRI. arXiv preprint arXiv:1706.00051,
2017. 1
N. Parikh, S. Boyd, et al. Proximal algorithms. Foundations
and Trends R in Optimization, 1(3):127–239, 2014. 2
Y. Romano, J. Isidoro, and P. Milanfar. RAISR: rapid and accurate image super resolution. IEEE Transactions on Computational Imaging, 3(1):110–125, 2017. 1, 7
J. Schlemper, J. Caballero, J. V. Hajnal, A. Price, and
D. Rueckert. A deep cascade of convolutional neural networks for MR image reconstruction. In Proceedings of the
25st Annual Meeting of ISMRM, Honolulu, HI, USA, 2017.
1, 2
C. K. Sonderby, J. Caballero, L. Theis, W. Shi, and F. Huszar.
Amortised MAP inference for image super-resolution. arXiv
preprint, arXiv:1610.04490, Oct. 2016. 1, 3, 7
J. Sun, H. Li, Z. Xu, et al. Deep ADMM-net for compressive
sensing MRI. In Advances in Neural Information Processing
Systems, pages 10–18, 2016. 1, 2
J. I. Tamir, F. Ong, J. Y. Cheng, M. Uecker, and M. Lustig.
Generalized magnetic resonance image reconstruction using
the berkeley advanced reconstruction toolbox. In ISMRM
Workshop on Data Sampling and Image Reconstruction, Sedona, 2016. 6
S. Wang, N. Huang, T. Zhao, Y. Yang, L. Ying, and D. Liang.
1d partial fourier parallel MR imaging with deep convolutional neural network. In Proceedings of the 25st Annual
Meeting of ISMRM, Honolulu, HI, USA, 2017. 1, 2
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to
structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 6
R. Yeh, C. Chen, T. Y. Lim, M. Hasegawa-Johnson, and
M. N. Do. Semantic image inpainting with perceptual and
contextual losses. arXiv preprint, arXiv:1607.07539, Jul.
2016. 1
H. Zhao, O. Gallo, I. Frosio, and J. Kautz. Loss functions for
image restoration with neural networks. IEEE Transactions
on Computational Imaging, 3(1), Mar. 2017. 1, 3
B. Zhu, J. Z. Liu, B. R. Rosen, and M. S. Rosen. Neural
network MR image reconstruction with AUTOMAP: Automated transform by manifold approximation. In Proceedings
of the 25st Annual Meeting of ISMRM, Honolulu, HI, USA,
2017. 1, 2, 3

