Auto-Encoding Total Correlation Explanation
  Advances in unsupervised learning enable reconstruction and generation of
samples from complex distributions, but this success is marred by the
inscrutability of the representations learned. We propose an
information-theoretic approach to characterizing disentanglement and dependence
in representation learning using multivariate mutual information, also called
total correlation. The principle of total Cor-relation Ex-planation (CorEx) has
motivated successful unsupervised learning applications across a variety of
domains, but under some restrictive assumptions. Here we relax those
restrictions by introducing a flexible variational lower bound to CorEx.
Surprisingly, we find that this lower bound is equivalent to the one in
variational autoencoders (VAE) under certain conditions. This
information-theoretic view of VAE deepens our understanding of hierarchical VAE
and motivates a new algorithm, AnchorVAE, that makes latent codes more
interpretable through information maximization and enables generation of richer
and more realistic samples.
