Plummer Autoencoders
Emanuele Sansone 1 Quoc-Tin Phan 1 Francesco G.B. De Natale 1

1. Introduction
Deep generative models, like autoregressive models (Larochelle & Murray, 2011), generative adversarial
networks (Goodfellow et al., 2014) and variational autoencoders (Kingma & Welling, 2013), represent very promising
research directions for solving the problem of density estimation. Each of these families have their own limitations.
The performance of existing autoregressive models based on
recurrent neural networks degrade as the number of features
(e.g the resolution of images) in data grows. Generative adversarial networks are difficult to train due to the mini-max
nature of the optimization problem and therefore it becomes
difficult to ensure global convergence of algorithms without
making very hard assumptions, like having networks with
infinite capacity. The performance of variational autoencoders are strongly dependent on the choice of posterior
density with direct consequence on the quality of the learnt
density.
In this work, we show that it is possible to cast the problem of density estimation as a minimization problem, thus
overcoming the difficulties encountered during the minimax optimization in generative adversarial networks. This
allows to exploit existing optimization routines for neural
networks and converge to global optimal solutions. Furthemore, we provide finite-sample bounds on the estimation
error of the true density. The whole framework is validated
through extensive experimental analysis on a variety of synthetic and real-world datasets by comparing the proposed

pZ (z)
l
qZ (z)

x

f (·)

z

qY (y)
g(·)

∼

pX (x)

∼

Estimating the true density in high-dimensional
feature spaces is a well-known problem in machine learning. This work shows that it is possible
to formulate the optimization problem as a minimization and use the representational power of
neural networks to learn very complex densities.
A theoretical bound on the estimation error is
given when dealing with finite number of samples.
The proposed theory is corroborated by extensive experiments on different datasets and compared against several existing approaches from
the families of generative adversarial networks
and autoencoder-based models.

∼

arXiv:1802.03505v1 [cs.LG] 10 Feb 2018

Abstract

y

Figure 1. Visualization of the encoding and decoding functions
with associated densities.

model against several state-of-the-art methods, including
generative adversarial networks, adversarial autoencoders
and variational autoencoders.
The remaining of the paper is organized as follows. In
Section 2, we provide the formulation of the optimization
problem, we analyze the properties of its objective function
and then provide a bound on the estimation error. In Section 3, we review the literature of recent generative models.
In Section 4, we discuss the experimental evaluation.

2. Approach
This section deals with the problem of density estimation.
The goal is to estimate the unknown density function pX (x),
whose support is defined by Ωx ⊂ Rd .
We start by providing the overall idea of learning the density
underlying the training data, then we formulate the optimization problem used in our approach and finally we analyze
the properties of the objective function and provide a bound
on the estimation error.
We consider two continuous functions f : Ωx → Ωz and
g : Ωz → Ωx , where Ωz ⊆ Rh and h is equal to the
intrinsic dimensionality of Ωx . Furthermore, we consider
that g(f (x)) = x for every x ∈ Ωx , namely that g is the left
inverse for f on domain Ωx . In this work, f and g are neural
networks parameterized by vectors γ and θ, respectively.
f is called the encoding function, taking a random input
x with density pX (x) and producing a random vector z
with density qZ (z), while g is the decoding function taking
z as input and producing the random vector y distributed
according to qY (y). Note that, pX (x) = qY (y), since
y = g(z) = g(f (x)) = x for every x ∈ Ωx . This is already
a density estimator, but it has the drawback that in general
qZ (z) and qY (y) cannot be written in closed form. Now,

Plummer Autoencoders

define pZ (z) an arbitrary density with support Ωz , that has
a closed form. Our goal is to guarantee that qZ (z) = pZ (z)
on the whole support, while maintaining g(f (x)) = x for
every x ∈ Ωx . This allows us to use the decoding function
as a generator and produce samples distributed according
to pX (x). Therefore, the problem of density estimation in a
high-dimensional feature space is converted into a problem
of estimation in a lower dimensional vector space, thus
overcoming the curse of dimensionality. Figure 1 provides
a graphical interpretation for these concepts.
The objective of our minimization problem is defined as
follows:
Z
L(f, g) =
kx − g(f (x))k2 pX (x)dx+
Ωx
Z Z
+λ
φ(z)φ(z0 )k(z, z0 )dzdz0
(1)
Ωz

The choice of the kernel function is of fundamental importance. In fact, authors in (Hochreiter & Obermayer, 2005)
have proved that gradient-based algorithms converge uniformly to the global solution of the second term in (1), only
for few choices of kernel function, e.g.pthe Plummer kernel, which is defined as k(z, z0 ) = 1/( kz − z0 k2 + )β ,
 > 0 is an arbitrary small parameter used to avoid the
singularity of k at z = z0 and β = h − 2. 1 The Gaussian
kernel does not satisfy this property and therefore cannot be
used to directly minimize the second term in (1). Practically
speaking, no local minima are present in the function space
when considering the Plummer kernel, whereas for other
kernels the problem of local minima may arise. In reality,
local minima are present when considering the parameter
space of neural networks. Nevertheless, it is theoretically
shown that for sufficiently powerful networks almost all
local minima are global minima (Nguyen & Hein, 2016).

Ωz

where φ(z)=pZ (z)−qZ (z), k(·, ·) is a positive definite kernel and λ > 0 is a factor used to weight the relative importance of the two addends.
Note that the first term in (1) reaches its global minimum
when the encoding and the decoding functions are invertible
on support Ωx , while the second term in (1) is globally optimal when qZ (z) equals pZ (z) (see Lemma 1). Therefore,
the global minimum of (1) satisfies our initial requirements
and the optimal solution corresponds to the case where
qX (x) = pX (x).

The result of (Hochreiter & Obermayer, 2005) holds also for
our objective, since the first addend in (1) is not affected by
the kernel choice. We use the extended version of the Plummer kernel (β = 1), which has the same above-mentioned
properties but leads to faster convergence (Unterthiner et al.,
2017).
It is important to mention that the integrals in (1) cannot be
computed exactly since pX (x) is unknown and qZ (z) is not
defined explicitly. As a consequence, we use the unbiased
estimate of (1) as a surrogate for optimization, namely:
X
b g)= 1
L(f,
kxi −g(f (xi ))k2 +
N
xi ∈Dx
(
X X
1
k(zi , zj )−
+λ
N (N − 1)

The following lemma proves that the global minimum of
the second term in (1) is achieved when qZ (z) equals pZ (z)
(proof in the supplementary material).
Lemma 1. Given k : Ωz × Ωz → R a symmetric positive
definite kernel, then:

zi ∈Dz zj ∈Dz
j6=i

−

(a) there exists a unique Hilbert space H of real-valued
functions over Ωz , for which k is a reproducing kernel.
H is therefore a Reproducing kernel Hilbert Space
(RKHS).
(b) For all ` ∈ H
Z Z
φ(z)φ(z0 )k(z, z0 )dzdz0 = MMD2 (pZ , qZ )
Ωz

Ωz

where

.
MMD(pZ , qZ ) = sup Ez∼pZ [`(z)]−Ez∼qZ [`(z)]
k`kH ≤1

is the maximum mean discrepancy between pZ (z) and
qZ (z).
(c) Let H be defined as in (b), then MMD(pZ , qZ ) = 0 if
and only if pZ (z) = qZ (z).

X
2
N (N − 1)

X

k(zi , zj )+

zi ∈Dz zj ∈D f
z
j6=i

X X
1
+
k(zi , zj )
N (N − 1)
f
f

)
(2)

zi ∈Dz zj ∈Dz
j6=i

N
f
N
where Dx ={xi }N
i=1 , Dz ={zi }i=1 and Dz ={f (xi )}i=1 are
two finite set of samples drawn from pX (x), pZ (z) and
qZ (z), respectively. Note that the first term in (2) corresponds to the reconstruction error on training data. Therefore, our model can be considered as an autoencoder. Based
on this fact and on the chosen kernel, we refer to our model
as Plummer autoencoder (PAE).

The following theorem provides a lower bound for the objective in (2) and a probabilistic bound on the estimation error
1

See Sec. 3 in (Hochreiter & Obermayer, 2005).

Plummer Autoencoders

b g) and L(f, g) (proof in the supplementary
between L(f,
material).
Theorem 1. Given the objective in (2), Ωz a compact set,
Ωx = [−M, M ]d , where M ∈ R+ , and a symmetric, continuous and positive definite kernel k : Ωz ×Ωz → R, where
0 ≤ k(z, z0 ) ≤ K for all z, z0 ∈ Ωz and K = k(z, z).
(a) There exists e
k : Ωz × Ωz → R such that k(z, z0 ) =
R
e
k(z, u)e
k(z0 , u)du.
Ωz
(b) The objective in (2) is equivalent to
X
e g) = 1
L(f,
kxi −g(f (xi ))k2 +
N
xi ∈Dx
(
Z
2
N
+λ
pZ,ek (u)−qZ,ek (u) du−
N −1 Ωz
)
X
2K
2
0
−
+
k(zi ,zi )
N −1 N (N −1)

In Algorithm 1, we provide the complete procedure of PAE.
Note that this requires to minimize only a single objective
function.
Algorithm 1 PAE, our proposed algorithm. All experiments
in the paper used the default values  = 0.0001, η = 0.0001,
β1 = 0.5, β2 = 0.9
Input: N mini-batch size, η learning rate.
Input: γ 0 inital parameter vector for f , θ 0 initial parameter vector for g.
repeat
Sample {xi }N
i=1 ∼ pX (x) a batch from training data.
Sample {zi }N
i=1 ∼ pZ (z) a batch from prior samples.
b
gγ ← ∇γ L(f ,g).
b ,g).
gθ ← ∇θ L(f
γ ← γ − η · Adam(γ, gγ , β1 , β2 ).
θ ← θ − η · Adam(θ, gθ , β1 , β2 ).
until γ, θ have not converged

zi ∈Dz
z0i ∈Dzf

P
where pZ,ek (u) = N1 zi ∈Dz e
k(zi , u) and qZ,ek (u) =
P
1
0
e
k(z
,
u)
are
the
kernel
density approximaf
i
z0i ∈Dz
N
tions of pZ (u) and q(u), respectively.
(c) Then, for any s, t > 0




N t2
b
Pr |L − L| > t + λs ≤2 exp −
+
8M 4 d2


bN/2cs2
+ 2 exp −
8K 2
Statement (b) of Theorem 1 provides an equivalent formulation of the objective in (2). This highlights the fact that2
e g) is always greater than − 2λK and therefore greater
L(f,
N −1
than zero when N is large.
e g) are the reconNote that the most influential terms in L(f,
struction error and the integral term, since the last addend
decays faster to zero as the number of samples increases.
Therefore, the model learns to reconstruct the traning data
with low error and learns to match qZ,ek (z) with pZ,ek (z).
Consequently, we can state that for sufficiently large N our
model generalizes.
Statement (c) provides a probabilistic bound on the estimab g) and L(f, g). The bound consists
tion error between L(f,
of two terms which both vanish when N is large. The first
one, due to the reconstruction error term, depends also on
the number of features d and on the side length M of the
hypercube Ωx , while the second one, due to the estimation
of the second integral in (1), depends on the maximum value
of the kernel function K.
2

e g) are zero.
When the second and the last addend in L(f,

3. Related work
The most promising research directions for generative models are autoregressive models, generative adversarial networks (GANs) and variational autoencoders (VAEs).
Autoregressive models (Larochelle & Murray, 2011) convert
the problem of density estimation into a sequence problem
based on the fact that the unknown density function can
be expressed as a product of conditional distributions. Recurrent neural networks (Oord et al., 2016) are generally
used to learn these conditional distributions. The main drawback is due to the difficulty of recurrent neural networks in
catching long-term dependencies. Therefore, performance
tend to degrade as the length of the sequences increase. Another limitation of these models is the absence of a latent
representation of data.
Generative adversarial networks (Goodfellow et al., 2014)
cast the problem of density estimation as a mini-max game
between two neural networks, namely a discriminator, that
tries to distinguish between true and generated samples, and
a generator, that tries to produce samples similar to the true
ones, to fool the discriminator. They have the reputation
of being difficult to train and also require careful design of
network architectures (Radford et al., 2015). Some of the
most known issues are (i) the problem of vanishing gradients (Arjovsky & Bottou, 2017), which happens when the
output of the discriminator is saturated, because true and
generated data are perfectly classified, and no more gradient
information is provided to the generator, (ii) the problem
of mode collapse (Metz et al., 2017), which happens when
the samples from the generator collapse to a single point
corresponding to the maximum output value of the discrimi-

Plummer Autoencoders

nator, and (iii) the problem of instability associated with the
failure of convergence, which is due to the intrinsic nature
of the mini-max problem.
Plethora of solutions exist in the literature and therefore we
focus only on the most relevant ones. To solve the problem
of vanishing gradients, the original paper of GANs (Goodfellow et al., 2014) proposes to use a different objective for the
generator, called the − log D alternative. The improved version from the same authors (Salimans et al., 2016) proposes
another objective for the generator which is based on feature
matching, namely comparing the mean statistics of true and
generated data on features extracted by the discriminator.
The work of (Durugkar et al., 2017) incorporates multiple
discriminators into the original framework, to ensure that
gradient continually flows. Arjowski et al. (Arjovsky & Bottou, 2017) analyzes the problem of vanishing gradients from
a theoretical perspective and provides a solution, which consists of adding instance noise during training to increase
the attraction force between the generated and the true data
manifold.
To solve the problem of mode collapse and more generally the problem of instability, authors in (Mescheder et al.,
2017b) introduce a regularizer in the optimization objective,
that acts as a form of consensus between the discriminator
and the generator and favors the convergence of training.
Nevertheless, only local convergence is guaranteed. Metz et
al. (Metz et al., 2017) propose an unrolled optimization of
the discriminator. The update of the generator is computed
by backpropagating the gradient through multiple updates
of the discriminator and generated samples are not any more
attracted to a single data point. Zhao et al. (Zhao et al.,
2017) formulate the problem differently. In particular, they
consider the discriminator as an energy function, assigning
low scores to samples on true data manifold and high scores
to samples on generated manifold, and use an autoencoder
as discriminator network. The authors show experimentally that the proposed model leads to better stability during
training. The work of (Karras et al., 2017) considers the
application of image generation and introduce a sort of curriculum in the training of GANs. A progressive growing
of the network capacity and of the image resolution allows
to stabilize the training. Their model is able to generate
images at a very high resolution. Nevertheless, the method
is heavily based on heuristics and more research effort is
required in order to formalize the approach and generalize
it to other domains.
GANs can be formulated also as a divergence minimization
problem. The seminal paper of Goodfellow et al. (Goodfellow et al., 2014) shows that, under the assumption of optimal
discriminator, the generator objective is equivalent to computing the Jensen Shannon divergence between the true and
the generated density. Authors (Nowozin et al., 2016) ex-

tend the analysis to a broader families of divergences, called
f −divergences. They compare the different measures from
this class and then provide some experimental insights on
which divergence to choose for natural images. The work
of (Arjovsky et al., 2017) defines some pathological examples, for which many divergences, including the Jensen
Shannon, yield to suboptimal solutions for the generator,
and therefore propose to use the Wasserstein distance. A
heuristic based on weight clipping is used to constrain the
critic3 to lie in the class of 1−Lipschitz functions. The
follow-up paper of (Gulrajani et al., 2017) substitute this
heuristic with a gradient penalty.
Another research direction for GANs consists on using integral probability metrics (Müller, 1997) as optimization objective. In particular, the maximum mean discrepancy (Gretton et al., 2008) can be used to measure the distance between
pX and qX and train the generator network. The general
problem is formulated in the following way:

inf sup Ex∼pX [f (x)] − Ex∼qX [f (x)]
g∈G f ∈F

In generative moment matching networks (Li et al., 2015;
Dziugaite et al., 2015) F is a RKHS, which is induced by
the Gaussian kernel. One drawback of these models is due
to the fact that no maximization is performed over F and
the resulting solutions are suboptimal. Another limitation is
due to the fact that the similarity scores associated with the
kernel function are directly computed in the sample space.
Therefore, the performance degrade as the dimensionality
of the feature space increases (Ramdas et al., 2015). The
work of (Li et al., 2017) introduces an encoding function to
represent data in a more compact way and distances are computed in the latent representation, thus solving the problem
of dimensionality. Authors (Mroueh et al., 2017) propose
to extend the maximum mean discrepancy and include also
covariance statistics to ensure better stability.
A common drawback of GANs is the lack of a latent representation of data. Authors (Chen et al., 2016) add an autoencoder network to the original framework for reconstructing
part of the latent code. The identical works of (Donahue
et al., 2017) and (Dumoulin et al., 2017) propose to add
an encoding function together with the generator and perform an adversarial game to ensure that the joint density
on the input/output of the generator agrees with the joint
density of the input/output of the encoder. They prove that
the optimal solution is achieved when the generator and the
encoder are invertible. In practice, they fail to guarantee the
convergence to that solution due to the adversarial nature
of the game. Authors (Srivastava et al., 2017) extend the
previous works by explicitly imposing the invertibility condition. They achieve this by adding a term to the generator
3
Namely, the discriminator in the traditional formulation of
GANs.

Plummer Autoencoders
Table 1. Test log-likelihood (LL) for different models on grid
dataset.
M ETHOD

LL

PAE
C OU GAN
WGAN
B I GAN
VEEGAN
VAE
AAE
AVB-AC

-4.8±0.4
-4.5±0.2
-4.1±0.1
-103.4±98.9
-26.9±6.8
-5.5±0.2
-7.5±0.4
-7.5±0.8

(a) True Data

(b) PAE

(c) CouGAN

(d) WGAN

(e) BiGAN

(f) VEEGAN

(g) VAE

(h) AAE

(i) AVB-AC

objective that computes the reconstruction error on the latent
space. Adversarial autoencoders (Makhzani et al., 2013) are
similar to these approaches with the only differences that
the estimation of the reconstruction error is performed in
the sample space, while the adversarial game is performed
only in the latent space.
It is important to mention that all of these works are based
on a mini-max problem, while our method solves a simple
minimization problem, for which it is possible to achieve
global convergence.
The most related work to ours is the one proposed by (Unterthiner et al., 2017). The authors optimize a similar objective and use the extended version of the Plummer kernel.
The computation of distances is performed directly in the
sample space and performance degrade as the number of
features increases. Another drawback is that their model
does not allow to infer a latent representation of data.
Variational autoencoders (Kingma & Welling, 2013) represent another family of implicit generative models. The
framework is based on maximizing the log-likelihood of
training data. In order to efficiently sample from the prior
and therefore estimate the log-likelihood, the authors introduce an inference network which gives information about
the most likely regions in the latent space to sample from.
Nevertheless, the performance of the model are strongly
dependent on the capability of the inference network to approximate the true posterior density. For this reason, the
work of (Mescheder et al., 2017a) proposes an adversarial
game to enforce this condition.

4. Experiments
Our proposed solution is compared against 7 methods,
namely Coulomb GAN (CouGAN) (Unterthiner et al.,
2017), which is the most closely related method, the improved version of Wasserstein GAN (WGAN) (Gulrajani
et al., 2017), Bidirectional GAN (BiGAN) (Donahue et al.,
2017), Variational Encoder Enhancement to GANs (VEEGAN) (Srivastava et al., 2017), Variational Autoencoders

Figure 2. Visualization of generated data from different models on
grid dataset.

(VAE) (Kingma & Welling, 2013), Adversarial Autoencoders (AAE) (Makhzani et al., 2013) and Adversarial Variational Bayes with Adaptive Contrast (AVB-AC) (Mescheder
et al., 2017a). Note that the first four competitors belong
to the family of generative adversarial networks, while the
last three are generative models based on autoencoders. We
make a huge effort in implementing and comparing all approaches on different synthetic and real-world datasets. The
code to replicate all experiments, including the ones for the
competitors are available at XXXXX.
4.1. Grid dataset
We start by comparing the approaches on a two-dimensional
dataset consisting of 25 isotropic Gaussians placed according to a grid (see Figure 2(a)), and call it the grid
dataset (Lim & Ye, 2017). The training dataset contains
500 samples generated from the true density.
Following the methodology of other works (see for example (Lim & Ye, 2017; Unterthiner et al., 2017)), we choose
fully connected MLPs with two hidden layers (128 neurons
each) in encoder, decoder and discriminator networks and
set h = 2. All models are trained for two million iterations
using Adam optimizer.
In our model, λ is chosen from the range {0.1, 1, 10} to
obtain the lowest value of objective at the end of training.4
For other models, we strictly follow the details of the origi4

In this set of experiments, λ = 10.

Plummer Autoencoders
Table 2. Test log-likelihood (LL) for different models on low dimensional embedding dataset.

(a) PAE

(b) WGAN

(e) VAE

(c) BiGAN

(f) AAE

(d) VEEGAN
(temp)

M ETHOD

LL

PAE
C OU GAN
WGAN
B I GAN
VEEGAN
VAE
AAE
AVB-AC

1224.4±76.3
FAILURE
848.9±534.4
-3940.3±3519.4
-7457806.0±4036262.3
-320.1±1081.6
1025.9±177.1
-965.4±1196.9

(g) AVB-AC

Figure 3. Detection of mode collapse through classification of generated samples on the low dimensional embedding dataset. The
x-axis of each plot represents modes (classes), while the y-axis
counts the number of generated samples for a given mode.

nal papers. It is important to mention that we observe poor
performance in VEEGAN and better results are obtained by
applying batch normalization to all hidden layers (results
without batch normalization and details of simulations are
available in the supplementary material).

use a classifier7 to count the number of samples generated by
the models for each mode. This evaluation procedure allows
to only detect the presence of mode collapse. It is important
to mention that this procedure can be fooled by specific
pathological cases, like memorization of training samples.
Therefore, we use log-likelihood on test data to assess the
quality of the learnt distribution. The other differences are in
the use of batch normalization for BiGAN and AAE, which
lead to better performance. We experience high instability
when training CouGAN and also convergence failures.

Models are evaluated qualitatively by visual inspecting generated samples and quantitatively by computing the loglikelihood on test data. To compute log-likelihood, we first
apply kernel density estimation using a Gaussian kernel on
105 generated samples5 and then evaluate the log-likelihood
on 104 test samples from the true distribution. Results are
averaged over 10 repetitions.

Figure 3 shows the histograms of generated samples obtained by all models. Note that BiGAN and VEEGAN are
affected by mode collapse. Table 2 provides log-likelihood
scores. Our model achieves the best performance, meaning
that it is able to better estimate the underlying true density.

Figure 2 shows samples generated by all models, while
Table 1 provides quantitative results. It is immediate to see
that BiGAN is affected by mode collapse, while AAE and
AVB-AC generate very noisy samples. VEEGAN achieves
very low performance in terms of log-likelihood since it
assigns high prior to only few modes. Our model, VAE and
CouGAN compare favourably with WGAN, which in turn
obtain the best performance.

The third dataset is created by stacking random digits from
the MNIST dataset on top of each other to produce colored
images (Srivastava et al., 2017). This creates a ground truth
density with 1000 classes.

4.2. Low dimensional embedding dataset
The second dataset consists of ten 10−d isotropic Gaussians
embedded in a 1000 dimensional vector space and call it
the low dimensional embedding dataset. We generate 500
samples from the true density to train all models.
The methodology is similar to the one of previous dataset. 6
The main difference is in the evaluation. Due to the difficulty
of visualizing samples in high dimensions, we propose to
5

Bandwidth is selected from a subset of values obtained from
the interval [10−3 , 101.5 ] by sampling 10 values equally spaced in
logarithmic scale.
6
In this set of experiments, λ = 1.

4.3. Stacked-MNIST

The methodology is similar to the one of previous dataset.8
We use the MLP network from (Cireşan et al., 2010) as
encoder, decoder and discriminator networks (see supplementary material for further details and simulations). We
train all models up to 1000 epochs with batch size equal
to 128. We visualize both generated samples and nearest
neighbors in the latent space, to understand the quality of the
learnt representation and see whether semantic consistency
is preserved in the neighborhood of samples. Similarly to
the previous case, we use batch normalization for BiGAN,
VEEGAN and AAE.
Figure 4 shows the generated samples of each model. Note
7

Consisting of a MLP with two hidden layers of 256 and 128
neurons each and trained on an infinitely large dataset sampled
from the true density for 1000 iterations and using Adam with
learning rate equal to 0.001.
8
In this set of experiments, λ = 0.1.

Plummer Autoencoders

(a) PAE

(b) CouGAN

(c) WGAN

(d) BiGAN

(e) VEEGAN

(f) VAE

(g) AAE

(h) AVB-AC

Figure 4. Visualization of generated samples from different models on Stacked-MNIST. The first column of each plot contains true
samples.

Table 3. Average number of training iterations per second for different models on Stacked-MNIST.
M ETHOD

(a) PAE

(b) BiGAN

(c) VEEGAN

(d) VAE

(e) AAE

(f) AVB-AC

Figure 5. Conditional generation of samples. The first column
of each plot contains true samples, while the other columns are
obtained by generating samples from the latent codes associated
to true data and perturbing the latent representation with additive
isotropic Gaussian noise (0.01 element variance).

that BiGAN and VEEGAN fail to learn the true distribution, while VAE and AVB-AC suffer from mode collapse.
CouGAN seems to generate more diverse samples, but with
very strong artifacts. Only PAE, AAE and WGAN are able

PAE
C OU GAN
WGAN
B I GAN
VEEGAN
VAE
AAE
AVB-AC

ITERS / SECS

40.4
29.0
7.1
14.7
15.3
48.4
10.4
25.8

to learn a good approximation of the true density. Figure 5
shows nearest neighbors for true data. It is worth to mention
that AAE does not preserve semantic consistency in the
representation. In fact, for a small perturbation of the latent
representation the output image can completely change its
semantic content. Instead our model is capable to fulfill this
property.
Finally, Table 3 shows an indicative measure of the training
rates achieved by all methods. It is quite evident that AAE
and our method outperform the other approaches in terms
of training speed.

Plummer Autoencoders

5. Conclusions
This work formulates the problem of density estimation as
a single minimization and provides a training algorithm for
neural networks to learn very complex densities.

References
Arjovsky, M. and Bottou, L. Towards principled methods for training generative adversarial networks. International Conference
on Learning Representations (ICLR), 2017.
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein Generative
Adversarial Networks. In International Conference on Machine
Learning (ICML), pp. 214–223, 2017.

Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progressive growing
of gans for improved quality, stability, and variation. arXiv
preprint arXiv:1710.10196, 2017.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes.
International Conference on Learning Representations (ICLR),
2013.
Larochelle, H. and Murray, I. The Neural Autoregressive Distribution Estimator. In International Conference on Artificial
Intelligence and Statistics (AISTATS), pp. 29–37, 2011.
Li, C. L., Chang, W. C., Cheng, Y., Yang, Y., and Póczos, B. MMD
GAN: Towards Deeper Understanding of Moment Matching
Network. In Advances in Neural Information Processing Systems (NIPS), pp. 2200–2210, 2017.

Chen, X., X.Chen, Duan, Y., Houthooft, R., Schulman, J.,
Sutskever, I., and Abbeel, P. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative
Adversarial Nets. In Advances in Neural Information Processing Systems (NIPS), pp. 2172–2180, 2016.

Li, Y., Swersky, K., and Zemel, R. Generative Moment Matching
Networks. In International Conference on Machine Learning
(ICML), pp. 1718–1727, 2015.

Cireşan, D. C., Meier, U., Gambardella, L. M., and Schmidhuber, J.
Deep, big, simple neural nets for handwritten digit recognition.
Neural Computation, 22:3207–3220, 2010.

Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., and Frey,
B. Adversarial autoencoders. International Conference on
Learning Representations (ICLR), 2013.

Donahue, J., Krähenbühl, P., and Darrell, T. Adversarial feature
learning. International Conference on Learning Representations
(ICLR), 2017.

Mescheder, L., Nowozin, S., and Geiger, A. Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks. In International Conference on
Machine Learning (ICML), pp. 2391–2400, 2017a.

Dumoulin, V., Belghazi, I., Poole, B., Lamb, A., Arjovsky, M.,
Mastropietro, O., and Courville, A. Adversarially learned inference. International Conference on Learning Representations
(ICLR), 2017.

Lim, J. H. and Ye, J. C.
arXiv:1705.02894, 2017.

Geometric gan.

arXiv preprint

Mescheder, L., Nowozin, S., and Geiger, A. The Numerics of
GANs. In Advances in Neural Information Processing Systems
(NIPS), pp. 1823–1833, 2017b.

Durugkar, I., Gemp, I., and Mahadevan, S. Generative multiadversarial networks. International Conference on Learning
Representations (ICLR), 2017.

Metz, L., Poole, B., Pfau, D., and Sohl-Dickstein, J. Unrolled
generative adversarial networks. International Conference on
Learning Representations (ICLR), 2017.

Dziugaite, G. K., Roy, D. M., and Ghahramani, Z. Training
Generative Neural Networks via Maximum Mean Discrepancy
Optimization. In Uncertainty in Artificial Intelligence (UAI),
pp. 258–267, 2015.

Mroueh, Y., Sercu, T., and Goel, V. McGan: Mean and Covariance Feature Matching GAN. In International Conference on
Machine Learning (ICML), pp. 2527–2535, 2017.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., WardeFarley, D., Ozair, S., Courville, A., and Bengio, Y. Generative
Adversarial Nets. In Advances in Neural Information Processing
Systems (NIPS), pp. 2672–2680, 2014.

Müller, A. Integral probability metrics and their generating classes
of functions. Advances in Applied Probability, 29(2):429–443,
1997.
Nachman, A. Theory of reproducing kernels. Transactions of the
American Mathematical Society, 68(3):337–404, 1950.

Gretton, A., Borgwardt, K., Rasch, M., Schölkopf, B., and Smola,
A. A kernel method for the two sample problem. Technical
report, Max Planck Institute for Biological Cybernetics, 2008.

Nguyen, Q. and Hein, M. The Loss Surface of Deep and Wide
Neural Networks. In International Conference on Machine
Learning (ICML), pp. 2603–2612, 2016.

Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and
Courville, A. Improved Training of Wasserstein GANs. In
Advances in Neural Information Processing Systems (NIPS), pp.
5769–5779, 2017.

Nowozin, S., Cseke, B., and Tomioka, R. f-GAN: Training Generative Neural Samplers Using Variational Divergence Minimization. In Advances in Neural Information Processing Systems
(NIPS), pp. 271–279, 2016.

Hochreiter, S. and Obermayer, K. Optimal Kernels for Unsupervised Learning. In IEEE International Joint Conference on
Neural Networks (IJCNN 2005), pp. 1895–1899, 2005.

Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. Pixel Recurrent
Neural Networks. In International Conference on Machine
Learning (ICML), pp. 1747–1756, 2016.

Hoeffding, W. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association,
58(301):13–30, 1963.

Radford, A., Metz, L., and Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial
networks. arXiv preprint arXiv:1511.06434, 2015.

Plummer Autoencoders

Z

Ramdas, A., Reddi, S. J., Poczos, B., Singh, A., and Wasserman,
L. On the Decreasing Power of Kernel and Distance Based
Nonparametric Hypothesis Tests in High Dimensions. In AAAI
Conference on Artificial Intelligence, pp. 3571–3577, 2015.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford,
A., Chen, X., and Chen, X. Improved Techniques for Training
GANs. In Advances in Neural Information Processing Systems
(NIPS), pp. 2234–2242, 2016.
Srivastava, A., Valkoz, L., Russell, C., Gutmann, M. U., and Sutton,
C. VEEGAN: Reducing Mode Collapse in GANs Using Implicit Variational Learning. In Advances in Neural Information
Processing Systems (NIPS), pp. 3310–3320, 2017.
Unterthiner, T., Nessler, B., Klambauer, G., Heusel, M., Ramsauer,
H., and Hochreiter, S. Coulomb gans: Provably optimal nash
equilibria via potential fields. arXiv preprint arXiv:1708.08819,
2017.
Zhao, J., Mathieu, M., and LeCun, Y. Energy-based generative
adversarial network. International Conference on Learning
Representations (ICLR), 2017.

A. Proof of Lemma 1
Lemma 2. Given k : Ωz × Ωz → R a symmetric positive
definite kernel, then:
(a) there exists a unique Hilbert space H of real-valued
functions over Ωz , for which k is a reproducing kernel.
H is therefore a Reproducing kernel Hilbert Space
(RKHS).
(b) For all ` ∈ H
Z Z
φ(z)φ(z0 )k(z,z0 )dzdz0 = MMD2 (pZ , qZ )
Ωz

Ωz

where

.
MMD(pZ , qZ ) = sup Ez∼pZ [`(z)]−Ez∼qZ [`(z)]
k`kH ≤1

is the maximum mean discrepancy between pZ (z) and
qZ (z).
(c) Let H be defined as in (b), then MMD(pZ , qZ ) = 0 if
and only if pZ (z) = qZ (z).

Z

pZ (z0 )qZ (z)k(z,z0 )dzdz0 +

−
Ωz

Z
+
Z

Ωz

Z

Ωz

q(z0 )qZ (z)k(z,z0 )dzdz0

Ωz

Z

pZ (z)pZ (z0 )hr(z), r(z0 )iH dzdz0 −

=
Ωz

Ωz

Z

Z

pZ (z)q(z0 )hr(z), r(z0 )iH dzdz0 −

−
ZΩz ZΩz
−
ZΩz ZΩz
+
Ωz

pZ (z0 )qZ (z)hr(z), r(z0 )iH dzdz0 +
q(z0 )qZ (z)hr(z), r(z0 )iH dzdz0

Ωz

= hEz∼pZ [r(z)], Ez0 ∼pZ [r(z0 )])iH −
− hEz∼pZ [r(z)], Ez0 ∼q [r(z0 )])iH −
− hEz0 ∼pZ [r(z0 )], Ez∼q [r(z)])iH +
+ hEz0 ∼q [r(z0 )], Ez∼q [r(z)])iH

(3)

Note that the second equality in (3) follows from the fact that
k(z,z0 ) = hr(z), r(z0 )iH for a unique r ∈ H,9 where h·,·iH
.
is the inner product of H. If we define µpZ = Ez∼pZ [r(z)]
.
10
and µq = Ez∼q [r(z)], then (3) can be rewritten in the
following way:
J = hµpZ ,µpZ iH − hµpZ ,µq iH − hµpZ ,µq iH + hµq ,µq iH
= hµpZ − µq ,µpZ − µq iH
= kµpZ − µq k2H

(4)

Notice that
D
µpZ − µq E
kµpZ −µq kH = µpZ − µq ,
kµpZ − µq kH H
n
o
= sup hµpZ − µq , `iH
k`kH ≤1

= sup

n
Ez∼pZ [hr(z), `iH ]−

k`kH ≤1

o
Ez∼q [hr(z), `iH ]

= sup Ez∼pZ [`(z)] − Ez∼q [`(z)]
k`kH ≤1

= MMD(pZ , qZ )
Proof. (a) follows directly from the Moore-Aronszajn theorem (Nachman, 1950).
Now we prove statement R(b).R For the sake of notation
.
compactness, define J = Ωz Ωz φ(z)φ(z0 )k(z,z0 )dzdz0 .
Therefore,
Z Z
J=
pZ (z)pZ (z0 )k(z,z0 )dzdz0 −
Ωz Ωz
Z Z
−
pZ (z)q(z0 )k(z,z0 )dzdz0 −
Ωz

Ωz

Substituting this result into (4) concludes the proof of the
statement.
Statement (c) is equivalent to Theorem 3 in (Gretton et al.,
2008).
9
This is a classical result due to the Riesz representation theorem.
10
Their existence can be guaranteed assuming that kµpZ k2H <
∞ and kµq k2H < ∞. In other words, Ez,z0 ∼pZ [k(z,z0 )] < ∞
and Ez,z0 ∼q [k(z,z0 )] < ∞.

Plummer Autoencoders

B. Proof of Theorem 1

−

=

zi ∈Dz zj ∈Dz

X

k(zi ,zj )+

zi ∈Dz zj ∈D f
z
j6=i

)
X X
1
k(zi ,zj )
+
N (N −1)
f
f
zi ∈Dz zj ∈Dz
j6=i

=

1 X
kxi −g(f (xi ))k2 +
N
xi ∈Dx
(
N
1 X X
+λ
k(zi ,zj )−
N −1 N 2
zi ∈Dz zj ∈Dz
j6=i

X

k(zi ,zj )+

zi ∈Dz zj ∈Dzf

+

1 X X
k(zi ,zj )−
N2
f
f
zi ∈Dz zj ∈Dz

1 X
2 X
− 2
k(zi ,zi ) + 2
k(zi ,z0i )−
N
N
zi ∈Dz

zi ∈Dz
z0i ∈Dzf

)
1 X
k(zi ,zi )
− 2
N
f
zi ∈Dz

1 X
kxi −g(f (xi ))k2 +
=
N
xi ∈Dx
(
N
1 X X
+λ
k(zi ,zj )−
N −1 N 2
zi ∈Dz zj ∈Dz

−

Proof. Property (a) can be proved using the Mercer’s theorem [see Theorem 1 in (Hochreiter & Obermayer, 2005)].
Now we prove property (b):
X
b ,g) = 1
L(f
kxi −g(f (xi ))k2 +
N
xi ∈Dx
(
X X
1
+λ
k(zi ,zj )−
N (N −1)
X
2
N (N −1)

1 X
kxi −g(f (xi ))k2 +
N
xi ∈Dx
(
N
1 X X
+λ
k(zi ,zj )−
N −1 N 2
2 X
− 2
N

P
where pZ,ek (u) = N1 zi ∈Dz e
k(zi ,u) and qZ,ek (u) =
P
1
0
e
z0i ∈Dzf k(zi ,u) are the kernel density approximaN
tions of pZ (u) and q(u), respectively.

−

)

zi ∈Dz zj ∈Dz
j6=i

zi ∈Dz
z0i ∈Dzf

zi ∈Dz zj ∈Dz
j6=i

k(zi ,zj )+

1 X X
+ 2
k(zi ,zj )
N
f
f

(b) The objective in (2) is equivalent to
X
e g) = 1
kxi −g(f (xi ))k2 +
L(f,
N
xi ∈Dx
(
Z
2
N
+λ
pZ,ek (u)−qZ,ek (u) du−
N −1 Ωz
)
X
2K
2
0
−
+
k(zi ,zi )
N −1 N (N −1)

(c) Then, for any s, t > 0




N t2
b
Pr |L − L| > t + λs ≤2 exp −
+
8M 4 d2


bN/2cs2
+ 2 exp −
8K 2

X

zi ∈Dz zj ∈D f
z
j6=i

Theorem 2. Given the objective in (2), Ωz a compact set,
Ωx = [−M, M ]d , where M ∈ R+ , and a symmetric, continuous and positive definite kernel k : Ωz ×Ωz → R, where
0 ≤ k(z,z0 ) ≤ K for all z,z0 ∈ Ωz and K = k(z,z).
(a) There exists e
k : Ωz × Ωz → R such that k(z,z0 ) =
R
e
e
k(z,u)k(z0 ,u)du.
Ωz

2 X
N2

2
N2

X

X

k(zi ,zj )+

zi ∈Dz zj ∈Dzf

1 X X
k(zi ,zj )−
N2
zi ∈Dzf zj ∈Dzf
)
2K
2 X
0
−
+ 2
k(zi ,zi )
N
N

+

zi ∈Dz
z0i ∈Dzf

=

1 X
kxi −g(f (xi ))k2 +
N
xi ∈Dx
(
Z
N
1 X X
e
+λ
k(zi ,u)e
k(zj ,u)du−
N −1 N 2
zi ∈Dz zj ∈Dz Ωz
Z
2 X X
e
k(zi ,u)e
k(zj ,u)du+
− 2
N
zi ∈Dz zj ∈Dzf Ωz
Z
1 X X
e
k(zi ,u)e
k(zj ,u)du−
+ 2
N
Ω
z
f
f
zi ∈Dz zj ∈Dz

Plummer Autoencoders

)

2K
2 X
−
+ 2
k(zi ,z0i )
N
N

−

zi ∈Dz
z0i ∈Dzf

2
1 X e
k(zj ,u) du−
N
f
zj ∈Dz

)
X
2K
2
0
1 X
−
+
k(zi ,zi )
kxi −g(f (xi ))k2 +
=
N −1 N (N −1)
N
z
∈D
i
z
xi ∈Dx
z0i ∈Dzf
(Z 
X
X
N
1
e g)
e
= L(f,
+λ
k(zi ,u)e
k(zj ,u)−
2
N −1
N
Ωz
zi ∈Dz zj ∈Dz
To prove property (c), we first derive the statistical bounds
2 X X e
− 2
k(zi ,u)e
k(zj ,u)+
for the two addends in (2), and then combine these results
N
zi ∈Dz zj ∈Dzf
in the final bound.

X
X
1
Consider the reconstruction error term in (2) and define
e
k(zi ,u)e
k(zj ,u) du−
+ 2
.
N
ξx = kx−g(f (x))k2 . Note that Ωx = [−M, M ]d and
zi ∈Dzf zj ∈Dzf
therefore ξx is bounded in the interval [0, 4M 2 d]. By con)
sidering ξx a random variable, we can apply the Hoeffding’s
2 X
2K
+ 2
k(zi ,z0i )
−
inequality [see Theorem 2 in (Hoeffding, 1963)] to obtain
N
N
zi ∈Dz
the following statistical bound:
0
f
zi ∈Dz


Z
1 X
1 X
=
kxi −g(f (xi ))k2 +
≥
t
≤
Pr
ξ
−
ξ
p
(x)dx
x
x X
N
N
xi ∈Dx
Ωx
x∈D
x
(Z 




X
1 X e
N
−2N 2 t2
N t2
e
+λ
k(z
,u)
k(z
,u)−
i
j
≤
2
exp
=
2
exp
−
(5)
N −1
N2
(4M 2 d)2 N
8M 4 d2
Ωz
zi ∈Dz
zj ∈Dz
X
2 X e
e
where t is an arbitrary small positive constant.
k(zi ,u)
k(zj ,u)+
− 2
N
zi ∈Dz
zj ∈Dzf
Note that the second addend in (2) is equivalent to the em
X
X
pirical unbiased estimate of MMD2 (pZ , qZ ) (Gretton et al.,
1
e
e
+ 2
k(zi ,u)
k(zj ,u) du−
2
\ (pZ , qZ ). Based on this
N
2008). We refer to it as MMD
zi ∈Dzf
zj ∈Dzf
observation, we can reuse the statistical bound provided in
)
2 X
2K
Theorem 14 of (Gretton et al., 2008), which is valid for any
0
+ 2
k(zi ,zi )
−
bounded positive definite kernel, namely:
N
N
zi ∈Dz


z0i ∈Dzf
2
2
X
\
1
Pr MMD (pZ , qZ ) − MMD (pZ , qZ ) > s ≤
=
kxi −g(f (xi ))k2 +
N


xi ∈Dx
bN/2cs2
(Z 
≤ 2 exp −
(6)
8K 2
N
1 X e
+λ
k(zi ,u)−
N −1
N
Ωz
zi ∈Dz
By combining (5) and (6), we obtain the following aggre2
gated bound:
1 X e
k(zj ,u) du−
−


Z
N
1 X
zj ∈Dzf
Pr
ξx −
ξx pX (x)dx ≥ t +
)
N
Ωx
2K
2 X
x∈D
x


−
+ 2
k(zi ,z0i )
2
N
N
2
\
zi ∈Dz
+Pr MMD (pZ , qZ ) − MMD (pZ , qZ ) > s ≤
z0i ∈Dzf




1 X
bN/2cs2
N t2
kxi −g(f (xi ))k2 +
=
≤ 2 exp −
+
2
exp
−
(7)
N
8M 4 d2
8K 2
xi ∈Dx
(
Z 
N
1 X e
Note that the sum of the two probabilities in (7) can be
+λ
k(zi ,u)−
lower-bounded in the following way:
N −1 Ωz N
zi ∈Dz


Z
1 X
Pr
ξx −
ξx pX (x)dx ≥ t +
N
Ωx
x∈Dx

Plummer Autoencoders


2
2
\
+Pr MMD (pZ , qZ ) − MMD (pZ , qZ ) > s ≥

Z
1 X
≥ Pr
ξx −
ξx pX (x)dx ≥ t ∪
N
Ωx
x∈Dx
i
2
2
\
∪ λ MMD (pZ , qZ )−MMD (pZ , qZ ) > λs ≥

Z
1 X
≥ Pr
ξx −
ξx pX (x)dx +
N
Ωx


Table 4. Network architectures.
M ETHOD

N ETWORK

N EURONS PER LAYER

PAE

E NCODER
D ECODER

2,128,128,2
2,128,128,2
2,128,128,2
2,128,128,1
2,128,128,2
2,128,128,1
2,128,128,2
2,128,128,2
4,128,128,1
2,128,128,254
254+1,128,128,2
256,128,128,1
2,128,128,4
2,128,128,2
2,128,128,2
2,128,128,2
2,128,128,1
2+64,128,128,2
2,128,128,2
4,128,128,1

C OU GAN

x∈Dx

WGAN

2

\ (pZ , qZ )−MMD2 (pZ , qZ ) >
+ λ MMD

> λs + t ≥

Z
1 X
≥ Pr
ξx −
ξx pX (x)dx+
N
Ωx

B I GAN
VEEGAN

x∈Dx

VAE

2

\ (pZ , qZ )−λMMD2 (pZ , qZ ) >
+ λMMD

> λs + t =


b
= Pr |L − L| > λs + t

AAE

(8)

AVB-AC

D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR

where the first inequality in (8) is obtained by applying the
union bound and by introducing λ; the second inequality is
obtained by replacing the union operator with the addition;
the third inequality is obtained by exploiting the triangle
inequality; the last equality follows from Lemma 1.
By combining (8) with (7), we get statement (c).
(a) CouGAN (ReLU) (b) CouGAN (ELU)

C. Experiments on Grid Dataset
Table 4 provides the details of the network architectures.
Hyperbolic tangent activations are used as output activations of encoders, while the outputs of decoders/generators
are linear. For all methods, we use ReLU hidden activations.
CouGAN uses ELU (Unterthiner et al., 2017), but we provide also results with ReLU. Furthermore, we show results
of VEEGAN with and without batch normalization (BN).
See Figure 6.

D. Experiments on Low Dimensional
Embedding Dataset
Table 5 provides the details of the network architectures.
Hyperbolic tangent activations are used as output activations of encoders, while the outputs of decoders/generators
are linear. For all methods, we use ReLU hidden activations. CouGAN uses ELU (Unterthiner et al., 2017), but
we provide also results with ReLU. Furthermore, we report
results of BiGAN, VEEGAN and AAE with and without
batch normalization (BN). See Table 6.

(c) VEEGAN w BN (d) VEEGAN w/o BN
Figure 6. Grid dataset.

E. Experiments on Stacked-MNIST
Table 7 provides the details of the network architectures. Hyperbolic tangent activations are used as output activations of
encoders, while the outputs of decoders/generators are sigmoid (standard logistic activations). For all methods, we use
ReLU hidden activations. CouGAN uses ELU (Unterthiner
et al., 2017), but we provide also results with ReLU. Furthermore, we report results of BiGAN, VEEGAN and AAE
with and without batch normalization (BN). See Figure 7.

Plummer Autoencoders

(a) CouGAN (ReLU)

(b) CouGAN (ELU)

(c) BiGAN w BN

(d) BiGAN w/o BN

(e) VEEGAN w BN

(f) VEEGAN w/o BN

(g) AAE w BN

(h) AAE w/o BN

Figure 7. Grid dataset.

Table 5. Network architectures.
M ETHOD

N ETWORK

N EURONS PER LAYER

PAE

E NCODER
D ECODER

1000,128,128,10
10,128,128,1000
10,128,128,1000
1000,128,128,1
10,128,128,1000
1000,128,128,1
1000,128,128,10
1000,128,128,10
1010,128,128,1
1000,128,128,10
10+1,128,128,1000
1010,128,128,1
1000,128,128,20
10,128,128,1000
1000,128,128,10
10,128,128,1000
10,128,128,1
1000+64,128,128,10
10,128,128,1000
1010,128,128,1

C OU GAN
WGAN
B I GAN
VEEGAN
VAE
AAE
AVB-AC

D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR

Table 6. Test log-likelihood (LL) for different models on low dimensional embedding dataset.
M ETHOD
C OU GAN (ELU)
C OU GAN (R E LU)
B I GAN W BN
B I GAN W / O BN
VEEGAN W BN
VEEGAN W / O BN
AAE W BN
AAE W / O BN

LL
FAILURE
FAILURE
-3940.3±3519.4
-55528.9±45685.8
-157286274.0±1036296.3
-7457806.0±4036262.3
1025.9±177.1
-44627.7±39387.2

Plummer Autoencoders

Table 7. Network architectures.
M ETHOD

N ETWORK

N EURONS PER LAYER

PAE

E NCODER
D ECODER

2352,2500,2000,1500,1000,500,10
10,500,1000,1500,2000,2500,2352
10,500,1000,1500,2000,2500,2352
2352,2500,2000,1500,1000,500,1
10,500,1000,1500,2000,2500,2352
2352,2500,2000,1500,1000,500,1
2352,2500,2000,1500,1000,500,10
10,500,1000,1500,2000,2500,2352
2362,2500,2000,1500,1000,500,1
2352+1,2500,2000,1500,1000,500,10
10,500,1000,1500,2000,2500,2352
2362,2500,2000,1500,1000,500,1
2352,2500,2000,1500,1000,500,20
10,500,1000,1500,2000,2500,2352
2352,2500,2000,1500,1000,500,10
10,500,1000,1500,2000,2500,2352
10,2500,2000,1500,1000,500,1
2352+64,2500,2000,1500,1000,500,10
10,500,1000,1500,2000,2500,2352
2362,2500,2000,1500,1000,500,1

C OU GAN
WGAN
B I GAN
VEEGAN
VAE
AAE
AVB-AC

D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR
E NCODER
D ECODER
D ISCRIMINATOR

