Title: Nonlinear Information Bottleneck,
Abstract: Information bottleneck [IB] is a technique for extracting information in some
`input' random variable that is relevant for predicting some different 'output'
random variable. IB works by encoding the input in a compressed 'bottleneck
variable' from which the output can then be accurately decoded. IB can be
difficult to compute in practice, and has been mainly developed for two limited
cases: (1) discrete random variables with small state spaces, and (2)
continuous random variables that are jointly Gaussian distributed (in which
case the encoding and decoding maps are linear). We propose a method to perform
IB in more general domains. Our approach can be applied to discrete or
continuous inputs and outputs, and allows for nonlinear encoding and decoding
maps. The method uses a novel upper bound on the IB objective, derived using a
non-parametric estimator of mutual information and a variational approximation.
We show how to implement the method using neural networks and gradient-based
optimization, and demonstrate its performance on the MNIST dataset.