Denoising Adversarial Autoencoders
Antonia Creswell

Anil Anthony Bharath

BICV
Imperial College London

BICV
Imperial College London

arXiv:1703.01220v4 [] 4 Jan 2018

Email: ac2211@ic.ac.uk

Abstract—Unsupervised learning is of growing interest because
it unlocks the potential held in vast amounts of unlabelled data to
learn useful representations for inference. Autoencoders, a form
of generative model, may be trained by learning to reconstruct
unlabelled input data from a latent representation space. More
robust representations may be produced by an autoencoder
if it learns to recover clean input samples from corrupted
ones. Representations may be further improved by introducing
regularisation during training to shape the distribution of the
encoded data in the latent space. We suggest denoising adversarial
autoencoders, which combine denoising and regularisation, shaping the distribution of latent space using adversarial training.
We introduce a novel analysis that shows how denoising may
be incorporated into the training and sampling of adversarial
autoencoders. Experiments are performed to assess the contributions that denoising makes to the learning of representations
for classification and sample synthesis. Our results suggest that
autoencoders trained using a denoising criterion achieve higher
classification performance, and can synthesise samples that are
more consistent with the input data than those trained without
a corruption process.1

Modelling and drawing data samples from complex, highdimensional distributions is challenging. Generative models
may be used to capture underlying statistical structure from
real-world data. A good generative model is not only able to
draw samples from the distribution of data being modelled,
but should also be useful for inference.
Modelling complicated distributions may be made easier by
learning the parameters of conditional probability distributions
that map intermediate, latent, [2] variables from simpler distributions to more complex ones [4]. Often, the intermediate
representations that are learned can be used for tasks such as
retrieval or classification [25], [21], [27], [31].
Typically, to train a model for classification, a deep convolutional neural network may be constructed, demanding large
labelled datasets to achieve high accuracy [15]. Large labelled
datasets may be expensive or difficult to obtain for some
tasks. However, many state-of-the-art generative models can
be trained without labelled datasets [9], [25], [14], [12]. For
example, autoencoders learn a generative model, referred to
as a decoder, by recovering inputs from corrupted [31], [12],
[5] or encoded [14] versions of themselves.
Two broad approaches to learning state-of-the-art generative
models that do not require labelled training data include: 1)
1 This work has been submitted to the IEEE for possible publication.
Copyright may be transferred without notice, after which this version may
no longer be accessible.

Figure 1: Comparison of autoencoding models: Previous
works include Denoising Autoencoders (DAE) [5], [31], Variational Autoencoders (VAE) [14], Adversarial Autoencoders
(AAE) [21] and Denoising Variational Autoencoders (DVAE)
[12]. Our contributions are the DAAE and the iDAAE models.
Arrows in this diagram represent mappings implemented using
trained neural networks.

introduction of a denoising criterion [32], [5], [31] – where
the model learns to reconstruct clean samples from corrupted
ones; 2) regularisation of the latent space to match a prior
[14], [21]; for the latter, the priors take a simple form, such
as multivariate normal distributions.
The denoising variational autoencoder [12] combines both
denoising and regularisation in a single generative model.
However, introducing a denoising criterion makes the variational cost function – used to match the latent distribution
to the prior – analytically intractable [12]. Reformulation of
the cost function [12], makes it tractable, but only for certain
families of prior and posterior distributions. We propose using
adversarial training [9] to match the posterior distribution to
the prior. Taking this approach expands the possible choices
for families of prior and posterior distributions.
When a denoising criterion is introduced to an adversarial
autoencoder, we have a choice to either shape the conditional
distribution of latent variables given corrupted samples to
match the prior (as done using a variational approach [12]),

or to shape the full posterior conditional on the original data
samples to match the prior. Shaping the posterior distribution
over corrupted samples does not require additional sampling
during training, but trying to shape the full conditional distribution with respect to the original data samples does. We
explore both approaches, using adversarial training to avoid the
difficulties posed by analytically intractable cost functions.
Additionally, a model that has been trained using the posterior conditioned on corrupted data requires an iterative process
for synthesising samples, whereas using the full posterior
conditioned on the original data does not. Similar challenges
exist for the denoising VAE, but were not addressed by Im et
al.[12]. We analyse and address these challenges for adversarial autoencoders, introducing a novel sampling approach for
synthesising samples from trained models.
In summary, our contributions include: 1) Two types of denoising adversarial autoencoders, one which is more efficient
to train, and one which is more efficient to draw samples from;
2) Methods to draw synthetic data samples from denoising
adversarial autoencoders through Markov chain (MC) sampling; 3) An analysis of the quality of features learned with
denoising adversarial autoencoders through their application
to discriminative tasks.
I. BACKGROUND
A. Autoencoders
In a supervised learning setting, given a set of training
data, {(yi , xi )}N
i=1 we wish to learn a model, fψ (y|x) that
maximises the likelihood, Ep(y|x) fψ (y|x) of the true label,
y given an observation, x. In the supervised setting, there
are many ways to calculate and approximate the likelihood,
because there is a ground truth label for every training data
sample.
When trying to learn a generative model, pθ (x), in the
absence of a ground truth, calculating the likelihood of the
model under the observed data distribution, Ex∼p(x) pθ (x),
is challenging. Autoencoders introduce a two step learning
process, that allows the estimation, pθ (x) of p(x) via an
auxiliary variable, z. The variable z may take many forms
and we shall explore several of these in this section. The
two step process involves first learning a probabilistic encoder
[14], qφ (z|x), conditioned on observed samples, and a second
probabilistic decoder [14], pθ (x|z), conditioned on the auxiliary variables. Using the probabilistic encoder, we may form
a training dataset, {(zi , xi )}N
i=1 where xi is the ground truth
output for x ∼ p(x|zi ) with the input being zi ∼ qφ (z|xi ).
The probabilistic decoder, pθ (x|z), may then be trained on
this dataset in a supervised fashion. By sampling pθ (x|z)
conditioning on suitable z’s we may obtain a joint distribution,
pθ (x, z), which may be marginalised by integrating over all z
to obtain, to pθ (x).
In some situations the encoding distribution is chosen rather
than learned [5], in other situations the encoder and decoder
are learned simultaneously [14], [21], [12].

B. Denoising Autoencoders (DAEs)
Bengio et al. [5] treat the encoding process as a local
corruption process, that does not need to be learned. The
corruption process, defined as c(x̃|x) where x̃, the corrupted x
is the auxiliary variable (instead of z). The decoder, pθ (x|x̃),
is therefore trained on the data pairs, {(x̃i , xi )}N
i=1 .
By using a local corruption process (e.g. additive white
Gaussian noise [5]), both x̃ and x have the same number of
dimensions and are close to each-other. This makes it very
easy to learn pθ (x|x̃). Bengio et al. [5] shows how the learned
model may be sampled using an iterative process, but does not
explore how representations learned by the model may transfer
to other applications such as classification.
Hinton et al. [11] show that when auxiliary variables of an
autoencoder have lower dimension than the observed data, the
encoding model learns representations that may be useful for
tasks such as classification and retrieval.
Rather than treating the corruption process, c(x̃, x), as an
encoding process [5] – missing out on potential benefits of
using a lower dimensional auxiliary variable – Vincent et al.
[31], [32] learn an encoding distribution, qφ (z|x̃), conditional
on corrupted samples. The decoding distribution, pθ (x|z)
learns to reconstruct images from encoded, corrupted images,
see the DAEs in Figure 1. Vincent et al. [32], [31] show
that compared to regular autoencoders, denoising autoencoders
learn representations that are more useful and robust for
tasks such as classification. Parameters, φ and θ are learned
simultaneously by minimisng the reconstruction error for the
training set, {(x̃i , xi )}M
i=1 , which does not include zi . The
ground truth zi for a given x̃i is unknown. The form of the
distribution over z, to which x samples are mapped, pθ (z) is
also unknown - making it difficult to draw novel data samples
from the decoder model, pθ (x|z).
C. Variational Autoencoders
Variational autoencoders (VAEs) [14] specify a prior distribution, p(z) to which qφ (z|x) should map all x samples, by
formulating and maximising a variational lower bound on the
log-likelihood of pθ (x).
The variational lower bound on the log-likelihood of pθ (x)
is given by [14]:
log pθ (x) ≥ Ez∼qφ (z|x) [log pθ (x|z)]−KL[qφ (z|x)||p(z)] (1)
The pθ (x|z) term corresponds to the likelihood of a reconstructed x given the encoding, z of a data sample x.
This formulation of the variational lower bound does not
involve a corruption process. The term KL[qφ (z|x)||p(z)] is
the Kullback-Libeller divergence between qφ (z|x) and p(z).
Samples are drawn from qφ (z|x) via a re-parametrisation trick,
see the VAE in Figure 1.
If qφ (z|x) is chosen to be a parametrised multivariate
Gaussian, N (µφ (x), σφ (x)), and the prior is chosen to be a
Gaussian distribution, then KL[qφ (z|x)||p(z)] may be computed analytically. KL divergence may only be computed
analytically for certain (limited) choices of prior and posterior
distributions.

VAE training encourages qφ (z|x) to map observed samples
to the chosen prior, p(z). Therefore, novel observed data
samples may be generated via the following simple sampling
process: zi ∼ p(z), xi ∼ pθ (x|zi ) [14].
Note that despite the benefits of the denoising criterion
shown by Vincent et al. [31], [32], no corruption process was
introduced by Kingma et al. [14] during VAE training.
D. Denoising Variational Autoencoders
Adding the denoising criterion to a variational autoencoder
is non-trivial because the variational lower bound becomes
intractable.
Consider R the conditional probability density function,
q̃φ (z|x) = qφ (z|x̃)c(x̃|x)dx̃, where qφ (z|x̃) is the probabilistic encoder conditioned on corrupted x samples, x̃, and
c(x̃|x) is a corruption process. The variational lower bound
may be formed in the following way [12]:




pθ (x, z)
pθ (x, z)
≥ Eq̃φ (z|x) log
log pθ (x) ≥ Eq̃φ (z|x) log
qφ (z|x̃)
q̃φ (x|z)
If qφ (z|x̃) is chosen to be Gaussian, then in many cases
q̃φ (z|x) will be a mixture of Gaussians. If this is the case,
there is no analytical solution for KL[q̃φ (z|x)||p(z)] and so
the denoising variational lower bound becomes analytically
intractable. However there may still be an analytical solution
for KL[qφ (z|x̃)||p(z)]. The denoising variational autoencoder
therefore maximises Eq̃(x|z) log[ qpφθ (x,z)
(z|x̃) ]. We refer to the model
which is trained to maximise this objective as a DVAE, see
the DVAE in Figure 1. Im et al. [12] show that the DVAE
achieves lower negative variational lower bounds than the
regular variational autoencoder on a test dataset.
However, note that qφ (z|x̃) is matched to the prior, p(z)
rather than q̃φ (z|x). This means that generating novel samples
using pθ (z|x) is not as simple as the process of generating
samples from a variational autoencoder. To generate novel
samples, we should sample zi ∼ q̃φ (z|x), xi ∼ pθ (x|zi ),
which is difficult because of the need to evaluate q̃φ (z|x).
Im et al. [12] do not address this problem.
For both DVAEs and VAEs there is a limited choice of
prior and posterior distributions for which there exists an analytic solution for the KL divergence. Alternatively, adversarial
training may be used to learn a model that matches samples
to an arbitrarily complicated target distribution – provided
that samples may be drawn from both the target and model
distributions.
II. R ELATED W ORK
A. Adversarial Training
In adversarial training [9] a model gφ (w|v) is trained to
produce output samples, w that match a target probability
distribution t(w). This is achieved by iteratively training
two competing models, a generative model, gφ (w|v) and a
discriminative model, dχ (w). The discriminative model is fed
with samples either from the generator (i.e. ‘fake’ samples) or
with samples from the target distribution (i.e. ‘real’ samples),
and trained to correctly predict whether samples are ‘real’ or

‘fake’. The generative model - fed with input samples v, drawn
from a chosen prior distribution, p(v) - is trained to generate
output samples w that are indistinguishable from target w
samples in order to ‘fool’ [25] the discriminative model into
making incorrect predictions. This may be achieved by the
following mini-max objective [9]:
min max Ew∼t(w) [log dχ (w)] + Ew∼gφ (w|v) [log(1 − dχ (w))]
g

d

It has been shown that for an optimal discriminative model,
optimising the generative model is equivalent to minimising
the Jensen-Shannon divergence between the generated and target distributions [9]. In general, it is reasonable to assume that,
during training, the discriminative model quickly achieves near
optimal performance [9]. This property is useful for learning
distributions for which the Jensen-Shannon divergence may
not be easily calculated.
The generative model is optimal when the distribution
of generated samples matches the target distribution. Under
these conditions, the discriminator is maximally confused
and cannot distinguish ‘real’ samples from ‘fake’ ones. As
a consequence of this, adversarial training may be used to
capture very complicated data distributions, and has been
shown to be able to synthesise images of handwritten digits
and human faces that are almost indistinguishable from real
data [25].
B. Adversarial Autoencoders
Makhzani et al. [21] introduce the adversarial autoencoder
(AAE), where qφ (z|x) is both the probabilistic encoding
model in an autoencoder framework and the generative model
in an adversarial framework. A new, discriminative model,
dχ (z) is introduced. This discriminative model is trained
to distinguish between latent samples drawn from p(z) and
qφ (z|x). The cost function used to train the discriminator,
dχ (z) is:
Ldis = −

N −1
2N
1 X
1 X
[log dχ (zi )] −
[log(1 − dχ (zj ))]
N i=0
N
j=N

where zi=1...N −1 ∼ p(z) and zj=N...2N ∼ qφ (z|x) and N
is the size of the training batch.
Adversarial training is used to match qφ (z|x) to an arbitrarily chosen prior, p(z). The cost function for matching qφ (z|x)
to prior, p(z) is as follows:
Lprior =

N −1
1 X
[log(1 − dχ (zi ))]
N i=0

(2)

where zi=0...N −1 ∼ qφ (z|x) and N is the size of a training
batch. If both Lprior and Ldis are optimised, qφ (z|x) will be
indistinguishable from p(z).
In Makhzani et al.’s [21] adversarial autoencoder, qφ (z|x)
is specified by a neural network whose input is x and whose
output is z. This allows qφ (z|x) to have arbitrary complexity,
unlike the VAE where the complexity of qφ (z|x) is usually
limited to a Gaussian. In an adversarial autoencoder the

posterior does not have to be analytically defined because an
adversary is used to match the prior, avoiding the need to
analytically compute a KL divergence.
Makhzani et al. [21] demonstrate that adversarial autoencoders are able to match qφ (z|x) to several different priors,
p(z), including a mixture of 10 2D-Gaussian distributions.
We explore another direction for adversarial autoencoders, by
extending them to incorporate a denoising criterion.
III. D ENOISING A DVERSARIAL AUTOENCODER
We propose denoising adversarial autoencoders - denoising
autoencoders that use adversarial training to match the distribution of auxiliary variables, z to a prior distribution, p(z).
We formulate two versions of a denoising adversarial autoencoder which are trained to approximately maximise the
denoising variational lower bound [12]. In the first version,
we directly match the posterior q̃φ (z|x) to the prior, p(z)
using adversarial training. We refer to this as an integrating
Denoising Adversarial Autoencoder, iDAAE. In the second,
we match intermediate conditional probability distribution
qφ (z|x̃) to the prior, p(z). We refer to this as a DAAE.
In the iDAAE, adversarial training is used to bypass analytically intractable KL divergences [12]. In the DAAE, using
adversarial training broadens the choice for prior and posterior
distributions beyond those for which the KL divergence may
be analytically computed.
A. Construction
The distribution
of encoded data samples is given by
R
q̃φ (z|x) = qφ (z|x̃)c(x̃|x)dx̃ [12]. The distribution of decoded data samples is given by pθ (x|z). Both qφ (z|x) and
pθ (x|z) may be trained to maximise the likelihood of a
reconstructed sample,Pby minimising the reconstruction cost
N −1
function, Lrec = N1 i=0 log pθ (x|zi ) where the zi are obtained via the following sampling process xi=0...N −1 ∼ p(x),
x̃i ∼ c(x̃|xi ), zi ∼ qφ (z|x̃i ), and p(x) is distribution of the
training data.
We also want to match the distribution of auxiliary variables,
z to a prior, p(z). When doing so, there is a choice to match
either q̃φ (z|x) or qφ (z|x̃) to p(z). Each choice has its own
trade-offs either during training or during sampling.
1) iDAAE: Matching q̃φ (z|x) to a prior: In DVAEs there
is often no analytical solution for the KL divergence between
q̃φ (z|x) and p(z) [12], making it difficult to match q̃φ (z|x) to
p(z). Rather, we propose using adversarial training to match
q̃φ (z|x) to p(z), requiring samples to be drawn from q̃φ (z|x)
during training. ItR is challenging to draw samples directly
from q̃φ (z|x) = qφ (z|x̃)c(x̃|x)dx̃, but it is easy to draw
samples from qφ (z|x̃) and so q̃φ (z|x) may be approximated
PM
1
by M
i=1 qφ (z|x̃i ), x̃i=1...M ∼ c(x̃|x0 ), x0 ∼ p(x) where
x ∼ p(x) are samples from the training data, see Figure
1. Matching is achieved by minimising the following cost
function:
Lprior =

N −1
1 X
[log(1 − dχ (ẑi ))]
N i=0

PM
1
∼
where ẑi=0...N −1 = M
j=1 zi,j , zi=1...N,j=1...M
qφ (z|x̃i,j ), x̃i=1...N,j=1...M ∼ c(x̃|xi ), xi=0...N −1 ∼ p(x).
2) DAAE: Matching qφ (z|x̃) to a prior: Since drawing
samples from qφ (z|x̃) is trivial, qφ (z|x̃) may be matched
to p(z) via adversarial training. This is more efficient than
matching q̃φ (z|x) since a Monte-Carlo integration step (in Section III-A1) is not needed, see Figure 1. In using adversarial
training in place of KL divergence, the only restriction is
that we must be able to draw samples from the chosen prior.
Matching may be achieved by minimising the following loss
function:
Lprior =

N −1
1 X
[log(1 − dχ (zi ))]
N i=0

where zi=1...N −1 ∼ qφ (z|x̃i ).
Though more computationally efficient to train, there are
drawbacks when trying to synthesise novel samples from
pθ (x) if qφ (z|x̃) – rather than q̃φ (z|x) – is matched to the
prior. The effects of using a DAAE rather than an iDAAE
may be visualized by plotting the empirical distribution of
encodings of both data samples and corrupted data samples
with the desired prior, these are shown in Figure 2.
IV. S YNTHESISING NOVEL SAMPLES
In this section, we review several techniques used to draw
samples from trained autoencoders, identify a problem with
sampling DVAEs, which also applies to DAAEs, and propose
a novel approach to sampling DAAEs; we draw strongly on
previous work by Bengio et al. [4], [5].
A. Drawing Samples From Autoencoders
New samples may be generated by sampling a learned
pθ (x|z), conditioning on z drawn from a suitable distribution.
In the case of variational [14] and adversarial [21] autoencoders, the choice of this distribution is simple, because during
training the distribution of auxiliary variables is matched to a
chosen prior distribution, p(z). It is therefore easy and efficient
to sample both variational and adversarial autoencoders via the
following process: z ∼ p(z), x ∼ pθ (x|z) [14], [21].
The process for sampling denoising autoencoders is more
complicated. In the case where the auxiliary variable is a
corrupted image, x̃ [3], the sampling process is as follows:
x0 ∼ p(x), x̃0 ∼ c(x̃|x0 ), x1 ∼ pθ (x|x̃0 ) [5]. In the case
where the auxiliary variable is an encoding, [31], [32] the
sampling process is the same, with pθ (x|x̃) encompassing both
the encoding and decoding process.
However, since a denoising autoencoder is trained to reconstruct corrupted versions of its inputs, x1 is likely to be very
similar to x0 . Bengio et al. [5] propose a method for iteratively
sampling denoising autoencoders by defining a Markov chain
whose stationary distribution - under certain conditions - exists
and is equivalent, under certain assumption, to the training
data distribution. This approach is generalised and extended
by Bengio et al. [4] to introduce a latent distribution with no
prior assumptions on z.

Notice that our first sample is any real vector of dimension
a, where a is the dimension of the latent space. This Markov
chain has the transition operator:
Tθ,φ (z (t+1) |z (t) ) =
Z

(a) DAAE

(b) iDAAE

Figure 2: Compare how iDAAE and DAAE match encodings to the prior when trained on the CelebA dataset.
encoding refers to qφ (z|x), prior refers to the normal prior
p(z), encoded corrupted data refers to qφ (z|x̃) (a) DAAE:
Encoded corrupted samples match the prior, (b) iDAAE:
Encoded data samples match the prior.

We now consider the implication for drawing samples
from denoising adversarial autoencoders introduced in Section
III-A. By using the iDAAE formulation (Section III-A1) –
where q̃φ (z|x) is matched to the prior over z – x samples may
be drawn from pθ (x|z) conditioning on z ∼ p(z). However, if
we use the DAAE – matching qφ (z|x̃) to a prior – sampling
becomes non-trivial.
On the surface, it may appear easy to draw samples from
DAAEs (Section III-A2), by first sampling the prior, p(z) and
then sampling pθ (x|z).RHowever, the full posterior distribution
is given by q̃φ (z|x) = qφ (z|x̃)c(x̃|x)dx̃, but only qφ (z|x̃) is
matched to p(z) during training (See figure 2). The implication
of this is that, when attempting to synthesize novel samples
from pθ (x|z), drawing samples from the prior, p(z), is unlikely
to yield samples consistent with p(x). This will become more
clear in Section IV-B.

qφ (z (t+1) |x̃(t) )c(x̃(t) |x(t) )pθ (x(t) |z (t) )dxdx̃

We will now show that under certain conditions this transition operatorRdefines an ergodic Markov chain that converges
to P(z) = q̃φ (z|x)p(x)dx in the following steps: 1) We
will show that that there exists a stationary distribution P(z)
for z (0) drawn from a specific choice of initial distribution
(Lemma 1). 2) The Markov chain is homogeneous, because
the transition operator is defined by a set of distributions whose
parameters are fixed during sampling. 3) We will show that the
Markov chain is also ergodic, (Lemma 2). 4) Since the chain is
both homogeneous and ergodic there exists a unique stationary
distribution to which the Markov chain will converge [23].
Step 1) shows that one stationary distribution is P(z),
which we now know by 2) and 3) to be the unique stationary
distribution. So the Markov chain converges to P(z).
In this section, only, we use a change of notation, where the
training data probability distribution, previously represented as
p(x) is represented as P(x), this is to help make distinctions
between “natural system” probability distributions and the
learned distributions. Further, note that p(z) is the prior, while
the distribution required for sampling P(x|z) is P(z) such
that:
Z
Z
P(x) = P(x|z)P(z)dz ≈ pθ (x|z)P(z)dz.
(5)

Z
P(z) =

Z Z
q̃φ (z|x)P(x)dx =

qφ (z|x̃)c(x̃|x)dx̃P(x)dx.
(6)

B. Proposed Method For Sampling DAAEs
Here, we propose a method for synthesising novel samples
using trained DAAEs. In order to draw samples from pθ (x|z),
we need to be able to draw samples from q̃φ (z|x).
To ensure that we draw novel data samples, we do not want
to draw samples from the training data at any point during
sample synthesis. This means that we cannot use data samples
from our training data to approximately draw samples from
q̃φ (z|x).
Instead, similar to Bengio et al. [5], we formulate a Markov
chain, which we show has the necessary properties
to converge
R
and that the chain converges to P(z) = q̃φ (z|x)p(x)dx.
Unlike Bengio’s formulation, our chain is initialised with a
random vector of the same dimensions as the latent space,
rather than a sample drawn from the training set.
We define a Markov chain by the following sampling
process:
z (0) ∼ Ra ,
x̃(t) ∼ c(x̃|x(t) ),

(4)

t ≥ 0.

For proof see Appendix.
Lemma 2. The Markov chain defined by the transition operator, Tθ,φ (zt+1 |zt ) (4) is ergodic, provided that the corruption
process is additive Gaussian noise and that the adversarial
pair, qφ (z|x̃) and dχ (z) are optimal within the adversarial
framework.
For proof see Appendix.
Theorem 1. Assuming that pθ (x|z) is approximately equal to
P(x|z), and that the adversarial pair – qφ (z|x) and dχ (z) –
are optimal, the transition operator Tθ,φ (z (t+1) |z (t) ) defines a
Markov
chain whose unique stationary distribution is P(z) =
R
q̃φ (z|x)P(x)dx.
Proof. This follows from Lemmas 1 and 2.

x(t) ∼ pθ (x|z (t) ),
z (t+1) ∼ qφ (z|x̃(t) ),

Lemma 1. P(z) is a stationary distribution for the Markov
chain defined by the sampling process in (3).

(3)

This sampling method uncovers the distribution P(z) on
which samples drawn from pθ (x|z) must be conditioned in

order to sample pθ (x). Assuming pθ (x|z) ≈ P(x|z), this
allows us to draw samples from P(x).
For completeness, we would like to acknowledge that there
are several other methods that use Markov chains during the
training of autoencoders [2], [22] to improve performance.
Our approach for synthesising samples using the DAAE is
focused on sampling only from trained models; the Markov
chain sampling is not used to update model parameters.
V. I MPLEMENTATION
The analyses of Sections III and IV are deliberately general:
they do not rely on any specific implementation choice to
capture the model distributions. In this section, we consider
a specific implementation of denoising adversarial autoencoders and apply them to the task of learning models for
image distributions. We define an encoding model that maps
corrupted data samples to a latent space Eφ (x̃), and Rθ (z)
which maps samples from a latent space to an image space.
These respectively draw samples according to the conditional
probabilities qφ (z|x̃) and pθ (x|z). We also define a corruption
process, C(x), which draws samples according to c(x̃|x).
The parameters θ and φ of models Rθ (z) and Eφ (z) are
learned under an autoencoder framework; the parameters φ
are also updated under an adversarial framework. The models
are trained using large datasets of unlabelled images.
A. The Autoencoder
Under the autoencoder framework, Eφ (x) is the encoder
and Rθ (z) is the decoder. We used fully connected neural
networks for both the encoder and decoder. Rectifying Linear
Units (ReLU) were used between all intermediate layers to
encourage the networks to learn representations that capture
multi-modal distributions. In the final layer of the decoder
network, a sigmoid activation function is used so that the
output represents pixels of an image. The final layer of the
encoder network is left as a linear layer, so that the distribution
of encoded samples is not restricted.
As described in Section III-A, the autoencoder is trained to
maximise the log-likelihood of the reconstructed image given
the corrupted image. Although there are several ways in which
one may evaluate this log-likelihood, we chose to measure
pixel wise binary cross-entropy between the reconstructed
sample, x̂ and the original samples before corruption, x.
During training we aim to learn parameters φ and θ that
minimise the binary cross-entropy between x̂ and x. The
training process is summarised by lines 1 to 9 in Algorithm 1
in the Appendix.
The vectors output by the encoder may take any real values,
therefore minimising reconstruction error is not sufficient to
match either qφ (z|x̃) or q̃φ (z|x) to the prior, p(z). For this,
parameters φ must also be updated under the adversarial
framework.
B. Adversarial Training
To perform adversarial training we define the discriminator
dχ (z), described in Section II-A to be a fully connected neural

network, which we denote Dχ (z). The output of Dχ (z) is a
“probability” because the final layer of the neural network has
a sigmoid activation function, constraining the range of Dχ (z)
to be between (0, 1). Intermediate layers of the network have
ReLU activation functions to encourage the network to capture
highly non-linear relations between z and the labels, {‘real’,
‘fake’}.
How adversarial training is applied depends on whether
q̃φ (z|x) or qφ (z|x̃) is being fit to the prior p(z). zf ake refers
to the samples drawn from the distribution that we wish to
fit to p(z) and zreal , samples drawn from the prior, p(z).
The discriminator, Dχ (z), is trained to predict whether z’s are
‘real’ or ‘fake’. This may be achieved by learning parameters
χ that maximise the probability of the correct labels being
assigned to zf ake and zreal . This training procedure is shown
in Algorithm 1 on Lines 14 to 16.
Drawing samples, zreal , involves sampling some prior distribution, p(z), often a Gaussian. Now, we consider how to
draw fake samples, zf ake . How these samples are drawn
depends on whether qφ (z|x̃) (DAAE) is being fit to the prior
or q̃φ (z|x) (iDAAE) is being fit to the prior. Drawing samples,
zf ake is easy if qφ (z|x̃) is being matched to the prior, as these
are simply obtained by mapping corrupted samples though the
encoder: zf ake = Eφ (x̃).
However, if q̃(z|x) is being matched to the prior, we must
use Monte Carlo sampling to approximate zf ake samples (see
Section III-A1). The process for calculating zf ake is given by
Algorithm 2 in the Appendix, and detailed in Section III-A1.
Finally, in order to match the distribution of zf ake samples
to the prior, p(z), adversarial training is used to update
parameters φ while holding parameters χ fixed. Parameters
φ are updated to minimise the likelihood that Dχ (·) correctly
classifies zf ake as being ‘fake’. The training procedure is laid
out in lines 18 and 19 of Algorithm 1.
Algorithm 1 shows the steps taken to train an iDAAE. To
train a DAAE instead, all lines in Algorithm 1 are the same
except Line 11, which may be replaced by zf ake = Eφ (x̃).
C. Sampling
Although the training process for matching q̃φ (z|x) to p(z)
is less computationally efficient than matching qφ (z|x̃) to p(z),
it is very easy to draw samples when q̃φ (z|x) is matched to
the prior (iDAAE). We simply draw a random z (0) value from
p(z), and calculate x(0) = Rθ (z (0) ), where x(0) is a new
sample. When drawing samples, parameters θ and φ are fixed.
If qφ (z|x̃) is matched to the prior (DAAE), an iterative
sampling process is needed in order to draw new samples from
p(x). This sampling process is described in Section IV-B. To
implement this sampling process is trivial. A random sample,
z (0) is drawn from any distribution; the distribution does not
have to be the chosen prior, p(z). New samples, z (t) are
obtained by iteratively decoding, corrupting and encoding z (t) ,
such that z (t+1) is given by:
z (t+1) = Eφ (C(Rθ (z (t) ))).

In the following section, we evaluate the performance of
denoising adversarial autoencoders on three image datasets,
a handwritten digit dataset (MNIST) [18], a synthetic colour
image dataset of tiny images (Sprites) [26], and a complex
dataset of hand-written characters [17]. The denoising and
non-denoising adversarial autoencoders (AAEs) are compared
for tasks including reconstruction, generation and classification.
VI. E XPERIMENTS & R ESULTS
A. Code Available Online
We make our PyTorch [24] code available at the following
link: https://github.com/ToniCreswell/pyTorch DAAE 2 .
B. Datasets
We evaluate our denoising adversarial autoencoder on three
image datasets of varying complexity. Here, we describe the
datasets and their complexity in terms of variation within the
dataset, number of training examples and size of the images.
1) Datasets: Omniglot: The Omniglot dataset is a handwritten character dataset consisting of 1623 categories of
character from 50 different writing systems, with only 20
examples of each character. Each example in the dataset is
105-by-105 pixels, taking values {0,1}. The dataset is split
such that 19 examples from 964 categories make up the
training dataset, while one example from each of those 964
categories makes up the testing dataset. The 20 characters from
each of the remaining 659 categories make up the evaluation
dataset. This means that experiments may be performed to
reconstruct or classify samples from categories not seen during
training of the autoencoders.
2) Datasets: Sprites: The sprites dataset is made up of 672
unique human-like characters. Each character has 7 attributes
including hair, body, armour, trousers, arm and weapon type,
as well as gender. For each character there 20 animations
consisting of 6 to 13 frames each. There are between 120 and
260 examples of each character, however every example is in
a different pose. Each sample is 60-by-60 pixels and samples
are in colour. The training, validation and test datasets are split
by character to have 500, 72 and 100 unique characters each,
with no two sets having the same character.
3) Datasets: CelebA: The CelebA dataset consists of 250k
images of faces in colour. Though a version of the dataset with
tightly cropped faces exists, we use the un-cropped dataset. We
use 1000 samples for testing and the rest for training. Each
example has dimensions 64-by-64 and a set of labelled facial
attributes for example, ‘No Beard’, ‘Blond Hair’, ‘Wavy Hair’
etc. . This face dataset is more complex than the Toronto Face
Dataset used by Makhzani et al. [21] for training the AAE.
2 An older version of our code in Theano available at https://github.com/
ToniCreswell/DAAE with our results presented in iPython notebooks. Since
this is a revised version of our paper and Theano is no longer being supported,
our new experiments on the CelebA datasets were performed using PyTorch.

C. Architecture and Training
For each dataset, we detail the architecture and training
parameters of the networks used to train each of the denoising
adversarial autoencoders. For each dataset, several DAAEs,
iDAAEs and AAEs are trained. In order to compare models
trained on the same datasets, the same network architectures,
batch size, learning rate, annealing rate and size of latent code
is used for each. Each set of models were trained using the
same optimization algorithm. The trained AAE [21] models
act as a benchmark, allowing us to compare our proposed
DAAEs and iDAAEs.
1) Architecture and Training: Omniglot: The decoder, encoder and discriminator networks consisted of 6, 3 and 2
fully connected layers respectively, each layer having 1000
neurons. We found that deeper networks than those proposed
by Makhazni et al. [21] (for the MNIST dataset) led to better
convergence. The networks are trained for 1000 epochs, using
a learning rate 10−5 , a batch size of 64 and the Adam [13]
optimization algorithm. We used a 200D Gaussian for the prior
and additive Gaussian noise with standard deviation 0.5 for
the corruption process. When training the iDAAE, we use
M = 5 steps of Monte Carlo integration (see Algorithm 2
in the Appendix).
2) Architecture and Training: Sprites: Both the encoder and
discriminator are 2-layer fully connected neural networks with
1000 neurons in each layer. For the decoder, we used a 3-layer
fully connected network with 1000 neurons in the first layer
and 500 in each of the last layers, this configuration allowed
us to capture complexity in the data without over fitting. The
networks were trained for 5 epochs, using a batch size of
128, a learning rate of 10−4 and the Adam [13] optimization
algorithm. We used an encoding 200 units, 200D Gaussian for
the prior and additive Gaussian noise with standard deviation
0.25 for the corruption process. The iDAAE was trained with
M = 5 steps of Monte Carlo integration.
3) Architecture and Training: CelebA: The encoder and
decoder were constructed with convolutional layers, rather
than fully connected layers since the CelebA dataset is more
complex than the Toronto face dataset use by Makhzani et al.
[21]. The encoder and decoder consisted of 4 convolutional
layers with a similar structure to that of the DCGAN proposed
by Radford et al. [25]. We used a 3-layer fully connected
network for the discriminator. Networks were trained for
100 epochs with a batch size of 64 using RMSprop with
learning rate 10−4 and momentum of ρ = 0.1 for training the
discriminator. We found that using smaller momentum values
lead to more blurred images, however larger momentum values
prevented the network from converging and made training
unstable. When using Adam instead of RMSprop (on the
CelebA dataset specifically) we found that the values in the
encodings became very large, and were not consistent with the
prior. The encoding was made up of 200 units and we used a
200D Gaussian for the prior. We used additive Gaussian noise
for the corruption process. We experimented with different
noise level, σ between [0.1, 1.0], we found several values in
this range to be suitable. For our classification experiments

Samples may be synthesized using the decoder of a trained
iDAAE or AAE by passing latent samples drawn from the
prior through the decoder. On the other hand, if we pass
samples from the prior through the decoder of a trained DAAE,
the samples are likely to be inconsistent with the training data.
To synthesize more consistent samples using the DAAE, we
draw an initial z (0) from any random distribution – we use a
Gaussian distribution for simplicity3 – and decode, corrupt
and encode the sample several times for each synthesized
sample. This process is equivalent to sampling a Markov chain
where one iteration of the Markov chain includes decoding,
corrupting and encoding to get a z (t) after t iterations. The
z (t) may be used to synthesize a novel sample which we call,
x(t) . x(0) is the sample generated when z (0) is passed through
the decoder.
To evaluate the quality of some synthesized samples, we
calculated the log-likelihood of real samples under the model
[21]. This is achieved by fitting a Parzen window to a
number of synthesised samples. Further details of how the
log-likelihood is calculated for each dataset is given in the
Appendix F.
We expect initial samples, x(0) ’s drawn from the DAAE
to have a lower (worse) log-likelihood than those drawn from
the AAE, however we expect Markov chain (MC) sampling to
improve synthesized samples, such that x(t) for t > 0 should
have larger log-likelihood than the initial samples. It is not
clear whether x(t) for t > 0 drawn using a DAAE will be better than samples drawn form an iDAAE. The purpose of these
experiments is to demonstrate the challenges associated with
drawing samples from denoising adversarial autoencoders,
and show that our proposed methods for sampling a DAAE
and training iDAAEs allows us to address these challenges.
We also hope to show that iDAAE and DAAE samples are
competitive with those drawn from an AAE.
1) Sampling: Omniglot: Here, we explore the Omniglot
dataset, where we look at log-likelihood score on both a testing
and evaluation dataset. Recall (Section VI-B1) that the testing
dataset has samples from the same classes as the training
dataset and the evaluation dataset has samples from different
classes.
First, we discuss the results on the evaluation dataset. The
results, shown in Figure 3, are consistent with what is expected
of the models. The iDAAE out-performed the AAE, with a
3 which

happens to be equivalent to our choice of prior

Average performance
of all models in
Omniglot Eval Dataset

800

x(1) x(5)

iDAAE
x(0)

AAE
700

Average performance
of all models in
Omniglot Test Dataset

600

x(1) x(5)

iDAAE
500

400

300

AAE

x

(0)

}

D. Sampling DAAEs and iDAAEs

900

negative log-likelihood

we fixed σ = 0.25 and for synthesis from the DAAE, to
demonstrate the effect of sampling, we used σ = 1.0. For the
iDAAE we experimented with M = 2, 5, 20, 50. We found
that M < 5 (when σ = 1.0), was not sufficient to train an
iDAAE. By comparing histograms of encoded data samples to
histograms of the prior (see Figure 2), for an iDAAE trained
with a particular M value, we are able to see whether M is
sufficiently larger or not. We found M = 5 to be sufficiently
large for most experiments.

DAAE
Sampling

Figure 3: Omniglot log-likelihood of pθ (x) compared on the
testing and evaluation datasets. The training and evaluation
datasets have samples from different handwritten character
classes. All models were trained using a 200D Gaussian prior.
The training and testing datasets have samples from the same
handwritten character classes.

less negative (better) log-likelihood. The initial samples drawn
using the DAAE had more negative (worse) log-likelihood
values than samples drawn using the AAE. However, after
one iteration of MC sampling, the synthesized samples have
less negative (better) log-likelihood values than those from
the AAE. Additional iterations of MC sampling led to worse
results, possibly because synthesized samples tending towards
multiple modes of the data generating distribution, appearing
to be more like samples from classes represented in the
training data.
The Omniglot testing dataset consists of one example of
every category in the training dataset. This means that if
multiple iterations of MC sampling cause synthesized samples
to tend towards modes in the training data, the likelihood
score on the testing dataset is likely to increase. The results
shown in Figure 3 confirm this expectation; the log-likelihood
for the 5th sample is less negative (better) than for the 1st
sample. These apparently conflicting results (in Figure 3) –
whether sampling improves or worsens synthesized samples –
highlights the challenges involved with evaluating generative
models using the log-likelihood, discussed in more depth by
Theis et al. [30]. For this reason, we also show qualitative
results.
Figure 4(a) shows an set of initial samples (x(0) ) drawn
from a DAAE and samples synthesised after 9 iterations (x(9) )
of MC sampling in Figure 4(b), these samples appear to be
well vaired, capturing multiple modes of the data generating
distribution.
2) Sampling: Sprites: In alignment with expectation, the
iDAAE model synthesizes samples with higher (better) loglikelihood, 2122 ± 5, than the AAE, 2085 ± 5. The initial
image samples drawn from the DAAE model under-perform
compared to the AAE model, 2056 ± 5, however after just
one iteration of sampling the synthesized samples have higher
log-likelihood than samples from the AAE. Results also show

(a) x(0)

(b) x(5)

(c) x(20)

Figure 6: DAAE Face Samples σ = 1.0
(a) x(0)

(b) x(9)

Figure 4: Omniglot Markov chain (MC) sampling: (a) Initial
sample, x(0) and (b) Corresponding samples, x(9) after 9
iterations of MC sampling. The chain was initialized with
z (0) ∼ N (0, I).

that synthesized samples drawn using the DAAE after one
iteration of MC sampling have higher likelihood, 2261 ± 5,
than samples drawn using either the the iDAAE or AAE
models.
When more than one step of MC sampling is applied, the
log-likelihood decreases, in a similar way to results on the
Omniglot evalutation dataset, this may be related to how the
training, and test data are split, each dataset has a unique set
of characters, so combinations seen during training will not
be present in the testing dataset. These results further suggest
that MC sampling pushes synthesized samples towards modes
in the training data.
3) Sampling: CelebA: In Figure 5 we compare samples
synthesized using an AAE to those synthesized using an
iDAAE trained using M = 5 intergration steps. Figure 5(b)
show samples drawn from the iDAAE which improve upon
those drawn from the AAE model.
In Figure 6 we show samples synthesized from a DAAE
using the iterative approach described in section IV-B for M =
{0, 5, 20}. We see that the initial samples, x(0) have blurry
artifacts, while the final samples, x(20) are more sharp and
free from the blurry artifacts.

samples may be related to the sampling procedure, which
accoriding to theory presented by Alain et al. [1], would be
similar to taking steps toward the highest density regions of
the distribution (i.e. the mode), explaining why samples appear
to be quite similar.
When comparing DAAE or iDAAE samples to samples
from other generative models such as GANs [9] we may notice
that samples are less sharp. However, GANs often suffer from
‘mode collapse’ this is where all synthesised samples are very
similar, the iDAAE does not suffer mode collapse and does
not require any additional procedures to prevent mode collapse
[27]. Further, (vanilla) GANs do not offer an encoding model.
Other GAN variants such as Bi-GAN [6] and ALI [7] do
offer encoding models, however the fidelity of reconstruction
is very poor. The AAE, DAAE and iDAAE models are able
to reconstruct samples faithfully. We will explore fidelity of
reconstruction in the next section and compare to a state-of-art
ALI that has been modified to have improved reconstruction
fidelity, ALICE [19].
We conclude this section on sampling, by making the
following observations; samples synthesised using iDAAEs
out-performed AAEs on all datasets, where M = 5. It is
convenient, that relatively small M yields improvement, as
the time needed to train an iDAAE may increase linearly with
M . We also observed that initial samples synthesized using
the DAAE are poor and in all cases even just one iteration of
MC sampling improves image synthesis.
Finally, evaluating generated samples is challenging: loglikelihood is not always reliable [30], and qualitative analysis
is subjective. For this reason, we provided both quantitative
and qualitative results to communicate the benefits of introducing MC sampling for a trained DAAE, and the advantages
of iDAAEs over AAEs.
E. Reconstruction

(a) AAE (Previous work)

(b) iDAAE (Our work)

Figure 5: CelebA iDAAE samples (a) AAE (no noise) with
ρ = 0.1 and (c) iDAAE (with noise) with ρ = 0.1, σ = 0.25
and M = 5.
When drawing samples from iDAAE and DAAE models
trained on CelebA, a critical difference between the two
models emerges: samples synthesized using a DAAE have
good structure but appear to be quite similar to each other,
while the iDAAE samples have less good structure but appear
to have lots of variation. The lack of variation in DAAE

The reconstruction task involves passing samples from the
test dataset through the trained encoder and decoder to recover
a sample similar to the original (uncorrupted) sample. The
reconstruction is evaluated by computing the mean squared
error between the reconstruction and the original sample.
We are interested in reconstruction for several reasons. The
first is that if we wish to use encodings for down stream tasks,
for example classification, a good indication of whether the
encoding is modeling the sample well is to check the reconstructions. For example if the reconstructed image is missing
certain features that were present in the original images, it

Table I: Reconstruction: Shows the mean squared error for
reconstructions of corrupted test data samples. This table
server two purposes: (1) To demonstrate that in most cases
the DAAE and iDAAE are better able to reconstruct images
compared to the AAE. (2) To motivate why we are interested
in AAEs, as opposed to other GAN [9] related approaches,
by comparing reconstruction error on MNIST for a state of
the art GAN variant the ALICE [19], which was designed to
improve reconstruction fidelity in GAN-like models.
(a) Original

(b) Reconstructions

Figure 7: CelebA Reconstruction Error with an iDAAE

may be that this information is not preserved in the encoding.
The second reason is that checking sample reconstructions
is also a method to evaluate whether the model has overfit
to test samples. The ability to reconstruct samples not seen
during training suggests that a model has not overfit. The final
reason, is to further motivate AAE, DAAE and iDAAE models
as alternatives to GAN based models that are augmented with
encoders [19], for down stream tasks that require good sample
reconstruction. We expect that adding noise during training
would both prevent over fitting and encourage the model to
learn more robust representations, therefore we expect that the
DAAE and iDAAE would outperform the AAE.
1) Reconstruction: Omniglot: Table I compares reconstruction errors of the AAE, DAAE and iDAAE trained on the
Omniglot dataset. The reconstruction errors for both the
iDAAE and the DAAE are less than the AAE. The results
suggest that using the denoising criterion during training
helps the network learn more robust features compared to the
non-denoising variant. The smallest reconstruction error was
achieved by the DAAE rather than the iDAAE; qualitatively,
the reconstructions using the DAAE captured small details
while the iDAAE lost some. This is likely to be related to
the multimodal nature of q̃φ (z|x) in the DAAE compared to
the unimodal nature of q̃φ (z|x) in an iDAAE.
2) Reconstruction: Sprites: Table I shows reconstruction
error on samples from the sprite test dataset for models trained
on the sprite training data. In this case only the iDAAE model
out-performed the AAE and the DAAE performed as well as
the AAE.
3) Reconstruction: CelebA: Table I shows reconstruction
error on the CelebA dataset. We compare AAE, DAAE and
iDAAE models trained with momentum, ρ = 0.1, where the
DAAE and iDAAE have corruption σ = 0.1 and the iDAAE is
trained with M = 10 integration steps. We also experimented
with M = 5 however, better results were obtained using
M = 10. While the DAAE performs similarly well to the
AAE, the iDAAE outperforms both. Figure 7 shows examples
of reconstructions obtained using the iDAAE. Although the
reconstructions are slightly blurred, the reconstructions are
highly faithful, suggesting that facial attributes are correctly
encoded by the iDAAE model.

Model
AAE
DAAE
iDAAE
ALICE [19]

Omniglot

Sprite

MNIST4

CelebA

0.047
0.029
0.031
-

0.019
0.019
0.018
-

0.017
0.015
0.018
0.080

0.500
0.501
0.495

F. Classification
We are motivated to understand the properties of the representations (latent encoding) learned by the DAAE and iDAAE
trained on unlabeled data. A particular property of interest is
the separability, in latent space, between objects of different
classes. To evaluate separability, rather than training in a semisupervised fashion [21] we obtain class predictions by training
an SVM on top of the representations, in a similar fashion to
that of Kumar et al. [16].
1) Classification: Omniglot: Classifying samples in the
Omniglot dataset is very challenging: the training and testing
datasets consists of 946 classes, with only 19 examples of
each class in the training dataset. The 946 classes make up
30 writing systems, where symbols between writing systems
may be visually indistinguishable. Previous work has focused
on only classifying 5, 15 or 20 classes from within a single
writing system [28], [33], [8], [17], however we attempt to
perform classification across all 946 classes. The Omniglot
training dataset is used to train SVMs (with RBF kernels)
on encodings extracted from encoding models of the trained
DAAE, iDAAE and AAE models. Classification scores are
reported on the Omniglot evaluation dataset, (Table II).
Results show that the DAAE and iDAAE out-perform the
AAE on the classification task. The DAAE and iDAAE also
out-perform a classifier trained on encodings obtained by
applying PCA to the image samples, while the AAE does not,
further showing the benefits of using denoising.
We perform a separate classification task using only 20
classes from the Omniglot evaluation dataset (each class has 20
examples). This second test is performed for two key reasons:
a) to study how well autoencoders trained on only a subset
of classes can generalise as feature extractors for classifiers of
classes not seen during autoencoder training; b) to facilitate
performance comparisons with previous work [8]. A linear
SVM classifier is trained on the 19 samples from each of the
20 classes in the evaluation dataset and tested on the remaining
1 sample from each class. We perform the classification 20
times, leaving out a different sample from each class, in each
experiment. The results are shown in Table II. For comparison,

Table II: Omniglot Classification on all 964 Test Set Classes
and On 20 Evaluation Classes.
Model
AAE
DAAE
iDAAE
PCA
Random chance

Test Acc.

Eval Acc. %

18.36%
31.74%
34.02%
31.02%
0.11%

78.75 %
83.00%
78.25%
76.75%
5%

we also show classification scores when PCA is used as a
feature extractor instead of a learned encoder.
Results show that the the DAAE model out-performs the
AAE model, while the iDAAE performs less well, suggesting
that features learned by the DAAE transfer better to new tasks,
than those learned by the iDAAE. The AAE, iDAAE and
DAAE models also out-perform PCA.
2) Classification: CelebA: We perform a more extensive set
of experiments to evaluate the linear separability of encodings
learned on the celebA dataset and compare to state of the art
methods including the VAE [14] and the β-VAE5 . [10].
We train a linear SVM on the encodings of a DAAE (or
iDAAE) to predict labels for facial attributes, for example
‘Blond Hair’, ‘No Beard’ etc. . In our experiments, we
compare classification accuracy on 12 attributes obtained using
the AAE, DAAE and iDAAE compared to previously reported
results obtained for the VAE [14] and the β-VAE [10], these
are shown in Figure 8. The results for the VAE and β-VAE
were obtained using a similar approach to ours and were
reported by Kumar et al. [16]. We used the same hyper parameters to train all models and a fixed noise level of σ = 0.25,
the iDAAE was trained with M = 10. The Figure (8) shows
that the AAE, iDAAE and DAAE models outperform the VAE
and β-VAE models on most facial attribute categories.
To compare models more easily, we ask the question,
‘On how many facial attributes does one model out perform
another?’, in the context of facial attribute classification. We
ask this question for various combinations of model pairs, the
results are shown in Figure 9. Figures 9 (a) and (b), comparing
the AAE to DAAE and iDAAE respectively, demonstrate
that for some attributes the denoising models outperform the
non-denoising models. More over, the particular attributes for
which the DAAE and iDAAE outperform the AAE is (fairly)
consistent, both DAAE and iDAAE outperform the AAE on
the (same) attributes: ‘Attractive’, ‘Blond Hair’, ‘Wearing Hat’,
‘Wearing Lipstick’. The iDAAE outperforms on an additional
attribute, ‘Arched Eyebrows’.
There are various hyper parameters that may be chosen
to train these models; for the DAAE and iDAAE we may
choose the level of corruption and for the iDAAE we may
additionally choose the number of integration steps, M used
during training. We compare attribute classification results for
3 vastly different choices of parameter settings. The results
5 The β-VAE [10] weights the KL term in the VAE cost function with
β > 1 to encourge better organisation of the latent sapce, factorising the
latent encoding into interpretable, indepenant compoenents

Figure 8: Facial Attribute Classification Comparison of
classification scores for an AAE, DAAE, iDAAE compared
to the VAE [14] and β-VAE [10]. A Linear SVM classifier
is trained on encodings to demonstrate the linear separability
of representation learned by each model. The attribute classification values for the VAE and β-VAE were obtained from
Kumar et al. [16]

(a)

(b)

(c)

(d)

Figure 9: On how many facial attributes does one model
out perform another? For each chart, each portion shows the
number of facial attributes that each model out performs the
other model in the same chart.

are presented as a bar chart in Figure 10 for the DAAE.
Additional results for the iDAAE are shown in the Appendix
(Figure 15). These figures show that the models perform well
under various different parameter settings. Figure 10 suggests
that the model performs better with a smaller amount of
noise σ = {0.1, 0.25} rather than with σ = 1.0, however
it is important to note that a large amount of noise does not
‘break’ the model. These results demonstrate that the model
works well for various hyper parameters, and fine tuning is not
necessary to achieve reasonable results (when compared to the
VAE for example). It is possible that further fine tuning may
be done to achieve better results, however a full parameter
sweep is highly computationally expensive.
From this section, we may conclude that with the exception
of 3 facial attributes, AAEs and variations of AAEs are able to
outperform the VAE and β-VAE on the task of facial attribute

during training or sampling. In terms of log-likelihood on the
synthesized samples, an iDAAE using even a small number
of integration steps (M = 5) during training of an iDAAE
leads to better quality images being generated, and similarly
using even one step of sampling with a DAAE leads to better
generations.
Conflicting log-likelihood values of generated samples between testing and evaluation datasets means that these measurements are not a clear indication of how the number of
sampling iterations affects the visual quality of samples synthesized using a DAAE. In some cases it may be necessary to
visually inspect samples in order to assess effects of multiple
sampling iterations (Figure 4).
VII. C ONCLUSION

Figure 10: DAAE Robustness to hyper parameters

classification. This suggests that AAEs and their variants are
interesting models to study in the setting of learning linearly
separable encodings. We also show that for a specific set
of several facial attribute categories, the iDAAE or DAAE
performs better than the AAE. This consistency, suggests that
there are some specific attributes that the denoising variants
of the AAE learn better than the non-denoising AAE.
G. Trade-offs in Performance
The results presented in this section suggest that both
the DAAE and iDAAE out-perform AAE models on most
generation and some reconstruction tasks and suggest it is
sometimes beneficial to incorporate denoising into the training
of adversarial autoencoders. However, it is less clear which
of the two new models, DAAE or iDAAE, are better for
classification. When evaluating which one to use, we must
consider both the practicalities of training, and for generative
purposes, the practicalities – primarily computational load –
of each model.
The integrating steps required for training an iDAAE means
that it may take longer to train than a DAAE. On the other
hand, it is possible to perform the integration process in
parallel provided that sufficient computational resource is
available. Further, once the model is trained, the time taken
to compute encodings for classification is the same for both
models. Finally, results suggest that using as few as M = 5
integrating steps during training, leads to an improvement in
classification score. This means that for some classification
tasks, it may be worthwhile to train an iDAAE rather than a
DAAE.
For generative tasks, neither the DAAE nor the iDAAE
model consistently out-perform the other in terms of loglikelihood of synthesized samples. The choice of model may
be more strongly affected by the computational effort required

We propose two types of denoising autoencoders, where a
posterior is shaped to match a prior using adversarial training.
In the first, we match the posterior conditional on corrupted
data samples to the prior; we call this model a DAAE. In the
second, we match the posterior, conditional on original data
samples, to the prior. We call the second model an integrating
DAAE, or iDAAE, because the approach involves using Monte
Carlo integration during training.
Our first contribution is the extension of adversarial autoencoders (AAEs) to denoising adversarial autoencoders (DAAEs
and iDAAEs). Our second contribution includes identifying
and addressing challenges related to synthesizing data samples
using DAAE models. We propose synthesizing data samples
by iteratively sampling a DAAE according to a MC transition
operator, defined by the learned encoder and decoder of the
DAAE model, and the corruption process used during training.
Finally, we present results on three datasets, for three tasks
that compare both DAAE and iDAAE to AAE models. The
datasets include: handwritten characters (Omniglot [17]), a
collection of human-like sprite characters (Sprites [26]) and
a dataset of faces (CelebA [20]). The tasks are reconstruction,
classification and sample synthesis.
ACKNOWLEDGMENT
We acknowledge the Engineering and Physical Sciences
Research Council for funding through a Doctoral Training
studentship. We would also like to thank Kai Arulkumaran
for interesting discussions and managing the cluster on which
many experiments were performed. We also acknowledge Nick
Pawlowski and Martin Rajchl for additional help with clusters.
R EFERENCES
[1] G. Alain and Y. Bengio. What regularized auto-encoders learn from the
data-generating distribution. The Journal of Machine Learning Research,
15(1):3563–3593, 2014.
[2] P. Bachman and D. Precup. Variational generative stochastic networks
with collaborative shaping. In Proceedings of the 32nd International
Conference on Machine Learning, pages 1964–1972, 2015.
[3] Y. Bengio. Learning deep architectures for AI. Foundations and trends R
in Machine Learning, 2(1):1–127, 2009.
[4] Y. Bengio, E. Thibodeau-Laufer, G. Alain, and J. Yosinski. Deep
generative stochastic networks trainable by backprop. In Journal of
Machine Learning Research: Proceedings of the 31st International
Conference on Machine Learning, volume 32, 2014.

[5] Y. Bengio, L. Yao, G. Alain, and P. Vincent. Generalized denoising
auto-encoders as generative models. In Advances in Neural Information
Processing Systems, pages 899–907, 2013.
[6] J. Donahue, P. Krähenbühl, and T. Darrell. Adversarial feature learning.
arXiv preprint arXiv:1605.09782, 2016.
[7] V. Dumoulin, I. Belghazi, B. Poole, A. Lamb, M. Arjovsky, O. Mastropietro, and A. Courville. Adversarially learned inference. arXiv preprint
arXiv:1606.00704, 2016.
[8] H. Edwards and A. Storkey. Towards a neural statistician. arXiv preprint
arXiv:1606.02185, 2016.
[9] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Nets. In
Advances in Neural Information Processing Systems, pages 2672–2680,
2014.
[10] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick,
S. Mohamed, and A. Lerchner. beta-vae: Learning basic visual concepts
with a constrained variational framework. 2016.
[11] G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of
data with neural networks. science, 313(5786):504–507, 2006.
[12] D. J. Im, S. Ahn, R. Memisevic, Y. Bengio, et al. Denoising criterion
for variational auto-encoding framework. In Proceeding of the 31 AAAI
Conference on Artificial Intelligence, 2017.
[13] D. Kingma and J. Ba. Adam: A method for stochastic optimization.
In Proceedings of the 2015 International Conference on Learning
Representations (ICLR-2015), 2014.
[14] D. P. Kingma and M. Welling. Auto-encoding variational Bayes.
In Proceedings of the 2015 International Conference on Learning
Representations (ICLR-2015), 2014.
[15] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification
with deep convolutional neural networks. In Advances in neural
information processing systems, pages 1097–1105, 2012.
[16] A. Kumar, P. Sattigeri, and A. Balakrishnan. Variational inference of
disentangled latent concepts from unlabeled observations. arXiv preprint
arXiv:1711.00848, 2017.
[17] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level
concept learning through probabilistic program induction. Science,
350(6266):1332–1338, 2015.
[18] Y. LeCun, C. Cortes, and C. J. Burges. The mnist database of
handwritten digits, 1998.
[19] C. Li, H. Liu, C. Chen, Y. Pu, L. Chen, R. Henao, and L. Carin.
Alice: Towards understanding adversarial learning for joint distribution
matching. Neural Information Processing Systems (NIPS), 2017.
[20] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes
in the wild. In Proceedings of International Conference on Computer
Vision (ICCV), 2015.
[21] A. Makhzani, J. Shlens, N. Jaitly, and I. Goodfellow. Adversarial
autoencoders. arXiv preprint arXiv:1511.05644, 2015.
[22] A. Nguyen, J. Yosinski, Y. Bengio, A. Dosovitskiy, and J. Clune. Plug
& play generative networks: Conditional iterative generation of images
in latent space. arXiv preprint arXiv:1612.00005, 2016.
[23] H. U. of Technology. Markov chains and stochastic sampling. [Chapter
1: pg5: Theorem (Markov Chain Convergence)].
[24] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin,
A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in
pytorch. 2017.
[25] A. Radford, L. Metz, and S. Chintala. Unsupervised representation
learning with deep convolutional generative adversarial networks. In
International Conference on Learning Representations (ICLR) 2016,
2015.
[26] S. E. Reed, Y. Zhang, Y. Zhang, and H. Lee. Deep visual analogymaking. In Advances in Neural Information Processing Systems, pages
1252–1260, 2015.
[27] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and
X. Chen. Improved techniques for training GANs. arXiv preprint
arXiv:1606.03498, 2016.
[28] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap.
One-shot learning with memory-augmented neural networks. arXiv
preprint arXiv:1605.06065, 2016.
[29] P. Y. Simard, D. Steinkraus, J. C. Platt, et al. Best practices for
convolutional neural networks applied to visual document analysis. In
ICDAR, volume 3, pages 958–962. Citeseer, 2003.
[30] L. Theis, A. v. d. Oord, and M. Bethge. A note on the evaluation of
generative models. In Proceedings of the International Conference of
Learning Representations, 2015.

[31] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and
composing robust features with denoising autoencoders. In Proceedings
of the 25th International Conference on Machine Learning, pages 1096–
1103. ACM, 2008.
[32] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol.
Stacked denoising autoencoders: Learning useful representations in a
deep network with a local denoising criterion. Journal of Machine
Learning Research, 11(Dec):3371–3408, 2010.
[33] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching
networks for one shot learning. In Advances in Neural Information
Processing Systems, pages 3630–3638, 2016.

A PPENDIX A
P ROOFS

then P(xj |z) = 0 (and vice verse), which is a contradiction
to 1).

Lemma 1. P(z) is a stationary distribution for the Markov
chain defined by the sampling process in 3.

A PPENDIX B
E XAMPLES O F S YTHESISED S AMPLES

Proof. Consider the case where z (0) ∼ P(z). x(0) ∼
pθ (x|z (0) ) is from P(x), by equation 5. Following the sampling process, x̃(0) ∼ c(x̃|x(0) ), z (0) ∼ qφ (z|x̃(0) ), z (1) is
also from P(z), by equation 6. Similar to proof in Bengio
[4]. Therefore P(z) is a stationary distribution of the Markov
chain defined by 3.
Lemma 2. The Markov chain defined by the transition operator, Tθ,φ (zt+1 |zt ) (4)is ergodic, provided that the corruption
process is additive Gaussian noise and that adversarial pair,
qφ (z|x̃) and dχ (z) are optimal within the adversarial framework.
Proof. Consider X = {x : P(x) > 0}, X̃ = {x̃ : c(x̃|x) > 0}
and Z = {z : P(z) > 0}. Where Z ⊆ {z : p(z) > 0} and
X ⊆ X̃:
1) Assuming that pθ (x|z) is a good approximation of the
underlying probability distribution, P(x|z), then ∀xj ∼
P(x) ∃ zi ∼ P(z) s.t. pθ (xj |zi ) > 0.
2) Assuming that adversarial training has shapped the
distribution of qφ (z|x̃) to match the prior, p(z), then
∀zi ∼ p(z) ∃ x̃j s.t. qφ (zi |x̃j ) > 0. This holds because
if not all points in p(z) could be visited, qφ (z|x̃) would
not have matched the prior.
1) suggests that every point in X may be reached from a point
in Z and 2) suggests that every point in Z may be reached from
a point in X̃. Under the assumption that c(x̃|x) is an additive
Gaussian corruption process then x̃i is likely to lie within a
(hyper) spherical region around xi . If the corruption process is
sufficiently large such that (hyper) spheres of nearby x samples
overlap, for an xi and xi+m ∃ a set {xi+1 , ..., xi+m−1 } such
that, sup(c(x̃|xi )) ∩ sup(c(x̃|xi+1 )) 6= ∅, ∀i = 1, ..., (m − 1)
and where sup is the support. Then, it is possible to reach
any zi from any zj (including the case j = i). Therefore, the
chain is both irreducible and positive recurrent.
To be ergodic the chain must also be aperiodic: between
any two points xi and xj , there is a boundary, where x
values between xi and the boundary are mapped to zi , and
points between xj and the boundary are mapped to zj . By
applying the corruption process to x(t) = xi , followed by the
reconstruction process, there are always at least two possible
outcomes because we assume that all (hyper) spheres induced
by the corruption process overlap with at least one other
(hyper) sphere: either x̃(t) is not pushed over the boundary
and z (t+1) = zi remains the same, or x̃(t) is pushed over
the boundary and z (t+1) = zj moves to a new state. The
probability of either outcome is positive, and so there is
always more that one route between two points, thus avoiding
periodicity provided that for xi 6= xj , zi 6= zj . Consider, the
case where for xi 6= xj , zi = zj , then it would not be possible
to recover both xi and xj using P(x|z) and so if P(xi |z) > 0

Figure 11: Examples of synthesised samples: Examples of
randomly sytheisised data samples.
A PPENDIX C
A LGORITHM F OR T RAINING THE I DAAE
Algorithm 1: iDAAE : Matching q̃φ (z|x) to p(z)
1

2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

x = {x1 , x2 , xN −1 } ∼ p(x) #draw a batch of samples
from the training data
for k = 1 to NoEpoch do
x̃ = C(x) #corrupt all samples
z = Eφ (x̃) #encode all corrupted samples
x̂ = Rθ (z) #reconstruct
#minimise reconstruction cost
Lrec = −(x̂ log x + (1 − x̂) log(1 − x)).mean()
φ ← φ − α∇φ Lrec
θ ← θ − α∇φ Lrec
#match q̃φ (z|x) to p(z) using adversarial training
zf ake =approx z(x) #draw samples for q̃φ (z|x)
zreal ∼ p(z) #draw samples from prior p(z)
#train the discriminator
PN −1
Ldis = − N1 i=0 log Dχ (zreali )
P
N −1
− N1 i=0 log(1 − Dχ (zf akei ))
χ ← χ − a∇χ Ldis
#train the decoder to match the prior
PN −1
Lprior = N1 i=0 log(1 − Dχ (zf akei ))
φ ← φ − α∇φ
end

Algorithm 2: Drawing samples from q̃φ (z|x)
1
2
3
4
5
6
7
8
9
10

function: approx z({x1 , x2 , ...xN −1 })
for i = 0 to N − 1 do
zˆi = [ ]
for j = 1 to M do
x̃i,j = C(xi )
zi,j = Eφ (x̃j )
end
PM
1
zˆi .append( M
j=1 zj )
end
return ẑ = {ẑ1 , ẑ2 ...ẑN −1 }

A PPENDIX D
A LGORITHM F OR M ONTE C ARLO I NTEGRATION
A PPENDIX E
MNIST R ESULTS
A. Datasets: MNIST

Table III: Models trained on MNIST: 5 models are trained
on the MNIST dataset. Corruption indicates the standard deviation of Gaussian noise added during the corruption process,
c(x̃|x). Prior indicated the prior distribution imposed on the
latent space. M is the number of Monte Carlo integration steps
(see Algorithm 2 in Appendix) used during training - this
applies only to the iDAAE.
ID

Model

Corruption

Prior

M

1
2
3
4
5

AAE
DAAE
DAAE
iDAAE
iDAAE

0.0
0.5
0.5
0.5
0.5

10D Gaussian
10D Gaussian
10-GMM
10D Gaussian
10D Gaussian

5
25

Table IV: MNIST Reconstruction: Recon. shows the mean
squared error for reconstructions of corrupted test data samples
accompanied by the standard error. Corruption is the standard
deviation of the additive Gaussian noise used during training
and testing.
Model & Training

Recon.

The MNIST dataset consists of grey-scale images of handwritten digits between 0 and 9, with 50k training samples,
10k validation samples and 10k testing samples. The training,
validation and testing dataset have an equal number of samples
from each category. The samples are 28-by-28 pixels. The
MNIST dataset is a simple dataset with few classes and many
training examples, making it a good dataset for proof-ofconcept. However, because the dataset is very simple, it does
not necessarily reveal the effects of subtle, but potentially
important, changes to algorithms for training or sampling. For
this reason, we consider two datasets with greater complexity.

distribution over latent space may be matched to a mixture
of Gaussians - something that could not be achieved easily by
a VAE [14].

B. Architecture and Training: MNIST

C. Reconstruction: MNIST

For the MNIST dataset, we train a total of 5 models detailed
in Table III. The encoder, decoder and discriminator networks
each have two fully connected layers with 1000 neurons each.
For most models the size of the encoding is 10 units, and the
prior distribution that the encoding is being matched to is a
10D Gaussian. All networks are trained for 100 epochs on the
training dataset, with a learning rate of 0.0002 and a batch
size of 64. The standard deviation of the additive Gaussian
noise used during training is 0.5 for all iDAAE and DAAE
models.
An additional DAAE model is trained using the same
training parameters and networks as described above but with a
mixture of 10 2D Gaussians for the prior. Each 2D Gaussian
with standard deviation 0.5 is equally spaced with its mean
around a circle of radius 4 units. This results in a prior with
10 modes separated from each other by large regions of very
low probability. This model of the prior is very unrealistic, as
it assumes that MNIST digits occupy distinct regions of image
probability space. In reality, we may expect two numbers
that are similar to exist side by side in image probability
space, and for there to exist a smooth transition between
handwritten digits. As a consequence of this, this model is
intended specifically for evaluating how well the posterior

Table IV shows reconstruction error on samples from
the testing dataset, for each of the models trained on the
MNIST training dataset. Reconstruction error for the DAAE
and iDAAE trained with a 10D Gaussian prior do not outperform the AAE. However, the reconstruction task for the
AAE model was less challenging than that of the DAAE and
iDAAE models because samples were not corrupted before
the encoding step. The best model for reconstruction was
the iDAAE with M = 5, increasing M to 25 led to worse
reconstruction error, as reconstructions tended to appear more
like mean samples.
Reconstruction error for the DAAE trained using a 2D
mixture of 10 Gaussians under-performed compared to the rest
of the models. All reconstructions looked like mean images,
which may be expected given the nature of the prior.

Model

Corruption

Prior

M

Mean ± s.e.

AAE
DAAE
DAAE
iDAAE
iDAAE

0.0
0.5
0.5
0.5
0.5

10D Gaussian
10D Gaussian
2D 10-GMM
10D Gaussian
10D Gaussian

5
25

0.017±0.001
0.023±0.001
0.043±0.001
0.022±0.001
0.026±0.001

D. Sampling: MNIST
To calculate the log-likelihood of samples drawn from
DAAE, iDAAE and AAE models trained on the MNIST
dataset, a Parzen window is fitted to 1×104 generated samples
from a single trained model. The bandwidth for constructing
the Parzen window was selected by choosing one of 10
values, evenly space along a log axis between −1 and 0,

that maximises the likelihood on a validation dataset. The
log-likelihood for each model was then evaluated on the test
dataset.
Table V shows the log-likelihood on the test dataset for
samples drawn from models trained on the MNIST training
dataset. First, we will discuss models trained with a 10D
Gaussian prior. The best log-likelihood was achieved by the
iDAAE with M = 5. Synthesised samples generated by MC
sampling from the DAAE model caused the log-likelihood to
decrease. This may be because the samples tended towards
mean samples, dropping modes, causing the log-likelihood to
decrease. Initial samples and those obtained after 5 iterations
of sampling are shown in Figure 12. The samples in 12(d)
are clearer than in 12(c). Although mode-dropping is not
immediately apparent, note that the digit 4 is not present after
5 iterations.
Now, we consider the DAAE model trained with a 2D, 10GMM prior. Samples drawn from the prior are shown in Figure
13(a). The purpose of this experiment was to show the effects
of MC sampling, where the distribution from which initial
samples of z0 are drawn is significantly different to the prior.
Samples of z0 were drawn from a normal distribution and
passed through the decoder of the DAAE model to produce
initial image samples, see Figure 13(c). A further 9 steps of
MC sampling were applied to synthesise the samples shown
in Figure 13(d). As expected, the initial image samples do
not look like MNIST digits, and MC sampling improves
samples dramatically. Unfortunately however, many of the
samples appear to correspond to samples at modes of the data
distribution. In addition, several modes appear to be missing
from the model distribution. This may be attributed to the
nature of the prior, since we did not encounter this problem to
the same extent when using a 10D Gaussian prior (see Figure
12).
Synthesising MNIST samples is fairly trivial, since there are
many training examples and few classes. For this reason, it is
difficult to see the benefits of using DAAE or iDAAE models
compared to AAE models. Now, we focus on the two more
complex datasets, Omniglot and Sprites. We find that iDAAE
models and correctly sampled DAAE models may be used
to synthesise samples with higher log-likelihood than samples
synthesised using an AAE.
1) Classification: MNIST: The MNIST dataset consists of
10 classes, [0, 9], the classification task involves correctly
predicting a label in this interval. For the MNIST dataset, the
SVM classifier is trained on encoded samples from the MNIST
training dataset and evaluated on encoded samples from the
MNIST testing dataset, results are shown in Table VI.
First, we consider the results for DAAE, iDAAE and AAE
models trained with a 10D Gaussian prior. Classifiers trained
on encodings extracted from the encoders of trained DAAE,
iDAAE or AAE models out-performed classifiers trained on
PCA of image samples. Classifiers trained on the encodings
extracted from the encoders of learned DAAE and iDAAE
models out-performed those trained on the encodings extracted
from the encoders of the AAE model.

(a) Prior (one channel)

(b) Latent samples

(c) x(0)

(d) x(5)

Figure 12: MNIST Markov chain (MC) sampling: One
channel of the 10D Gaussian Prior a) One channel of the
10D Gaussian prior used to train the DAAE. b) Samples drawn
from qφ (z|x̃), projected in to 2D space, and colour coded by
the digit labels. c) Initial samples, x(0) generated by a normal
distribution. d) Corresponding samples after 5 iterations of
MC sampling. Notice how the highlighted “1” changes to a
“2” after 5 iterations of MC sampling.

(a) 2D Gaussian Mixture prior

(b) Latent samples

(c) x(0)

(d) x(9)

Figure 13: MNIST Markov chain (MC) sampling: a) 2D
Gaussian Mixture prior used to train a DAAE. b) Samples
drawn from qφ (z|x̃) projected into 2D space using PCA
and colour coded by the digit labels. c) Initial sample, x(0) ,
generated by a normal distribution. d) Corresponding samples
generated after 9 further iterations of MC sampling.

Table V: MNIST log-likelihood of pθ (x): To calculate the
log-likelihood of pθ (x), a Parzen window was fit to 104
generated samples and the mean likelihood was reported for
a testing data set. The bandwidth used for the Parzen window
was determined using a validation set. The training, test and
validation datasets had different samples.
Model & Training

Log-likelihood

Model

Corruption

Prior

M

x(0)

x(5)

AAE
DAAE
DAAE
iDAAE
iDAAE

0.0
0.5
0.5
0.5
0.5

10D Gaussian
10D Gaussian
2D 10-GMM
10D Gaussian
10D Gaussian

5
25

529
529
9
532
508

444
205
-

Table VI: MNIST Classification: SVM classifiers with RBF
kernels were trained on encoded MNIST training data samples.
The samples were encoded using the encoder of the trained
AAE, DAAE or iDAAE models. Classification scores are
given for the MNIST test dataset.
Model & Training
Model

Corruption

AAE
DAAE
DAAE
iDAAE
iDAAE
PCA + SVM

0.0
0.5
0.5
0.5
0.5

Prior

M

10D Gaussian
10D Gaussian
2D 10-GMM
10D Gaussian
10D Gaussian

5
25

Accuracy
%
95.19%
95.28%
73.81%
96.48%
96.24%
94.73%

The differences in classification score for each model on the
MNIST dataset are small; this might be because it is relatively
easy to classify MNIST digits with very high accuracy [29].
We turn now to a more complicated dataset, the Omniglot
dataset, to better show the benefits of using denoising when
training adversarial autoencoders.
A PPENDIX F
D ETAILS FOR CALCULATING LOG - LIKELIHOOD
A. Omniglot
To calculate log-likelihood of samples, a Parzen window
was fit to 1 × 103 synthesised samples, where the bandwidth
was determined on the testing dataset in a similar way to that
in Section E-D. The log-likelihood was evaluated on both the
evaluation dataset, and the testing dataset. To compute the loglikelihood on the of the testing dataset a Parzen window was
fit to a new set of synthesized samples, different to those used
to calculate the bandwidth. The results are shown in Figure 3.
B. Sprites
To calculate log-likelihood of samples, a Parzen window
was fit to 1×103 synthesized samples. The bandwidth was set
as previously described in Section E-D; we found the optimal
bandwidth to be 1.29.
I DAAE

A PPENDIX G
ROBUSTNESS TO HYPER PARAMETERS

Figure 14: iDAAE
Figure 15: Robustness to hyper parameters

