Title: Auto-Encoding Total Correlation Explanation,
Abstract: Advances in unsupervised learning enable reconstruction and generation of
samples from complex distributions, but this success is marred by the
inscrutability of the representations learned. We propose an
information-theoretic approach to characterizing disentanglement and dependence
in representation learning using multivariate mutual information, also called
total correlation. The principle of total Cor-relation Ex-planation (CorEx) has
motivated successful unsupervised learning applications across a variety of
domains, but under some restrictive assumptions. Here we relax those
restrictions by introducing a flexible variational lower bound to CorEx.
Surprisingly, we find that this lower bound is equivalent to the one in
variational autoencoders (VAE) under certain conditions. This
information-theoretic view of VAE deepens our understanding of hierarchical VAE
and motivates a new algorithm, AnchorVAE, that makes latent codes more
interpretable through information maximization and enables generation of richer
and more realistic samples.