Quantifying the similarity of information contained in probabilistic latent spaces
Information theory;representation learning;disentanglement
We found a bug in the calculation of the mutual information between full latent spaces (Fig 4), and the relevant results change significantly enough that we feel resubmission is necessary.
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
We compare the information content of probabilistic representation spaces, and use it to analyze VAE latent spaces and to perform ensemble learning.
@misc{
murphy2024quantifying,
title={Quantifying the similarity of information contained in probabilistic latent spaces},
author={Kieran A. Murphy and Sam Dillavou and Danielle Bassett},
year={2024},
url={https://openreview.net/forum?id=0K0hoNL9sx}
}
In contrast to point-based representation spaces, probabilistic representation spaces have a well-defined sense in which they compress information about a dataset.
When viewing representation spaces as communication channels, it becomes natural to ask about the similarity of information content of different representation spaces.
Starting with classic measures of similarity of hard clustering assignments, we propose a natural modification that generalizes to probabilistic representation spaces.
We also propose a practical route toward estimating the similarity measure based on fingerprinting a representation space with a sample of the dataset that is applicable when the transmitted information is only a handful of bits.
Equipped with the similarity measures, we build upon model centrality as a signature of unsupervised disentanglement by assessing ``channel centrality'' and finding information fragments that are repeatedly learned in VAE and InfoGAN ensembles.
Additionally, we evaluate the diversity of information content of the full latent space over the course of training for ensembles of models, and find a striking difference in homogeneity of information depending on the dataset.
Finally, we leverage the differentiability of the proposed method and perform ensemble learning with VAEs by boosting the information content of a set of weak learners incapable of representing the global structure of a dataset.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Kieran_A._Murphy1 ~Sam_Dillavou1 ~Danielle_Bassett1
Kieran A. Murphy Sam Dillavou Danielle Bassett
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Information theory representation learning disentanglement
I certify that there is no acknowledgement section in this submission for double blind review.
murphy|quantifying_the_similarity_of_information_contained_in_probabilistic_latent_spaces
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Quantifying the similarity of information contained in probabilistic latent spaces
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission