Title: Trimming the Independent Fat: Sufficient Statistics, Mutual Information, and Predictability from Effective Channel States,
Abstract: One of the most fundamental questions one can ask about a pair of random
variables X and Y is the value of their mutual information. Unfortunately, this
task is often stymied by the extremely large dimension of the variables. We
might hope to replace each variable by a lower-dimensional representation that
preserves the relationship with the other variable. The theoretically ideal
implementation is the use of minimal sufficient statistics, where it is
well-known that either X or Y can be replaced by their minimal sufficient
statistic about the other while preserving the mutual information. While
intuitively reasonable, it is not obvious or straightforward that both
variables can be replaced simultaneously. We demonstrate that this is in fact
possible: the information X's minimal sufficient statistic preserves about Y is
exactly the information that Y's minimal sufficient statistic preserves about
X. As an important corollary, we consider the case where one variable is a
stochastic process' past and the other its future and the present is viewed as
a memoryful channel. In this case, the mutual information is the channel
transmission rate between the channel's effective states. That is, the
past-future mutual information (the excess entropy) is the amount of
information about the future that can be predicted using the past. Translating
our result about minimal sufficient statistics, this is equivalent to the
mutual information between the forward- and reverse-time causal states of
computational mechanics. We close by discussing multivariate extensions to this
use of minimal sufficient statistics.