We introduce a new framework for unsupervised learning of representations based on a novel hierarchical decomposition of information. Intuitively, data is passed through a series of progressively fine-grained sieves. Each layer of the sieve recovers a single latent factor that is maximally informative about multivariate dependence in the data. The data is transformed after each pass so that the remaining unexplained information trickles down to the next layer. Ultimately, we are left with a set of latent factors explaining all the dependence in the original data and remainder information consisting of independent noise. We present a practical implementation of this framework for discrete variables and apply it to a variety of fundamental tasks in unsupervised learning including independent component analysis, lossy and lossless compression, and predicting missing values in data.
@cite_32: We define the relevant information in a signal @math as being the information that this signal provides about another signal @math . Examples include the information that face images provide about the names of the people portrayed, or the information that speech sounds provide about the words spoken. Understanding the signal @math requires more than just predicting @math , it also requires specifying which features of @math play a role in the prediction. We formalize this problem as that of finding a short code for @math that preserves the maximum information about @math . That is, we squeeze the information that @math provides about @math through a bottleneck' formed by a limited set of codewords @math . This constrained optimization problem can be seen as a generalization of rate distortion theory in which the distortion measure @math emerges from the joint statistics of @math and @math . This approach yields an exact set of self consistent equations for the coding rules @math and @math . Solutions to these equations can be found by a convergent re-estimation method that generalizes the Blahut-Arimoto algorithm. Our variational principle provides a surprisingly rich framework for discussing a variety of problems in signal processing and learning, as will be described in detail elsewhere.
@cite_11: Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.
Connections with ICA were discussed in Sec. and the relationship to InfoMax was discussed in Sec. . The information bottleneck (IB) @cite_32 is another information-theoretic optimization for constructing representations of data that has many mathematical similarities to the objective in Eq. , with the main difference being that IB focuses on supervised learning while ours is an unsupervised approach. Recently, the IB principle was used to investigate the value of depth in the context of supervised learning @cite_11 . The focus here, on the other hand, is to find an information-theoretic principle that justifies and motivates deep representations for unsupervised learning.