Title: Tackling Over-pruning in Variational Autoencoders,
Abstract: Variational autoencoders (VAE) are directed generative models that learn
factorial latent variables. As noted by Burda et al. (2015), these models
exhibit the problem of factor over-pruning where a significant number of
stochastic factors fail to learn anything and become inactive. This can limit
their modeling power and their ability to learn diverse and meaningful latent
representations. In this paper, we evaluate several methods to address this
problem and propose a more effective model-based approach called the epitomic
variational autoencoder (eVAE). The so-called epitomes of this model are groups
of mutually exclusive latent factors that compete to explain the data. This
approach helps prevent inactive units since each group is pressured to explain
the data. We compare the approaches with qualitative and quantitative results
on MNIST and TFD datasets. Our results show that eVAE makes efficient use of
model capacity and generalizes better than VAE.