A correspondence between thermodynamics and inference
Colin H. LaMont and Paul A. Wiggins

arXiv:1706.01428v4 [math.ST] 7 Apr 2018

Departments of Physics, Bioengineering and Microbiology, University of Washington, Box 351560.
3910 15th Avenue Northeast, Seattle, WA 98195, USA∗
We explore a natural analogy between Bayesian statistics and thermal physics in which sample size
corresponds to inverse temperature. This analogy motivates the definition of two novel statistical
quantities: a learning capacity and a Gibbs entropy. The analysis of the learning capacity, corresponding to the heat capacity in thermal physics, leads to a critical insight into why some models have
anomalously good learning performance. The mechanism is a statistical analogue of the failure of the
equipartition theorem formula for the heat capacity. We explore the properties of the learning capacity
in a number of examples, including a sloppy model. We also propose that the Gibbs entropy provides
a natural device for counting distinguishable distributions in the context of Bayesian inference. This
insight results in a new solution to a long-standing problem in Bayesian inference: the definition of an
objective or uninformative prior. We use the Gibbs entropy to define a generalized principle of indifference (GPI) in which every distinguishable model is assigned equal a priori probability. This approach
both resolves a number of long standing inconsistencies in objective Bayesian inference and unifies
several seemingly unrelated Bayesian methods with the information-based paradigm of inference.

I.

INTRODUCTION

Despite significant advances in learning algorithms,
fundamental questions remain about the mechanisms
of learning and the relationships between learning algorithms. How does model complexity affect learning? Why do some models have anomalously good
learning performance? In this paper, we explore the
phenomenology of learning by exploiting a correspondence between Bayesian inference and statistical mechanics. This correspondence has been previously described by Jaynes, Balasubramanian, and many others
[1–6] and many methods from statistical physics have
been adapted to statistics [7–17]. We extend this correspondence by using the canonical bridge between statistical mechanics and thermodynamics to compute the
standard thermodynamic potentials and properties of
a learning system. The correspondence identifies two
novel statistical quantities, a learning capacity and the
Gibbs entropy. These quantities generate new insights
into the mechanism of learning and new learning algorithms.
The analysis of a novel learning capacity (corresponding to the heat capacity) reveals an interesting connection between the Akaike Information Criterion (AIC)
of information-based inference and the equipartition
theorem in statistical mechanics [18]. In addition,
the learning capacity also provides new insights into
the mechanism of learning. It has long been known
that some high-dimensional models learn anomalously
well. These models have been termed sloppy [19]. We
demonstrate that the learning capacity both provides
a natural definition for the sloppiness phenomenon as
well as providing a mechanism: a statistical analogue
of the well known freeze-out mechanism of statistical
mechanics. We hypothesize that this mechanism is re-

∗

pwiggins@uw.edu; http://mtshasta.phys.washington.edu/

sponsible for the anomalously high predictive performance of many high-dimensional models.
We also propose that the Gibbs entropy provides
a natural device for determining model multiplicity,
i.e. counting indistinguishable distributions in the context of statistical inference. This interpretation allows
us to define a generalized principle of indifference (GPI)
for selecting a prior in absence of a priori information
about parameters or models. The GPI unifies a number of known, but seemingly unconnected objective
Bayesian methods, while also providing an algorithm
applicable to small sample size and singular models
where existing approaches fail. The GPI also resolves
a number of troubling anomalies in objective Bayesian
analysis, providing a natural resolution to the LindleyBartlett paradox in which larger models are automatically rejected.
The paper is organized as follows: In Sec. II, we define the correspondence. In Sec. III, we compute the
thermodynamics potentials and properties of inference
in the analytically tractable large-sample-size limit and
use these results to deduce the statistical meaning of
each quantity. In Sec. IV, we compute the learning capacity and GPI prior for a number of example analyses
to demonstrate that the results in Sec. III generalize beyond the normal model.

II.

DEFINING THE CORRESPONDENCE

We assume that a true parameter value θ0 is drawn
from a known prior distribution $(θ). We observe N
samples xN ≡ {x1 , ..., xN } which are distributed like
q(x|θ0 ):
Xi ∼ q(·|θ0 ),

(1)

where we use capital X to denote random variables
and the symbol ∼ to denote distributed like. For simplicity, we will assume that the observations are inde-

2
Thermodynamics

Statistics

Quantity:

Interpretation:

β = T −1

Inverse temperature

θ

State variables/vector
N

Quantity:

Interpretation:

↔

N

Sample size

↔

θ

Model parameters
N

Quenched disorder

↔

X

EX (θ)

State energy

↔

ĤX (θ)

Cross entropy estiamtor

E0

Disorder-averaged ground state energy

↔

H0

Shannon entropy

ρ(θ)

Density of states

↔

$(θ)

Prior

Z

Partition function

↔

Z

X

Z

−1

ρ exp −βEX

F = −β

−1

log Z

U = ∂β βF
C=

−β 2 ∂β2 βF
2

↔

Normalized Boltzmann weight

Evidence
N

Posterior

$(θ|X )
−1

Free energy

↔

F = −N

Average energy

↔

U = ∂N N F

↔

Heat capacity

↔

Gibbs entropy

S = β ∂β F

Observations

C=

log Z

Minus-log-prediction

2
−N 2 ∂N
NF
2

S = N ∂N F

Minus-log-evidence
Learning capacity
Statistical Gibbs entropy

TABLE I. Thermodynamic-Bayesian correspondence. The top half of the table lists the correspondences that can be determined
directly from the definition of the marginal likelihood as the partition function. The lower half of the table lists the implied
thermodynamic expressions and their existing or proposed statistical interpretation.

pendent and identically distributed, but the approach
can be generalized.
The correspondence between statistical physics and
Bayesian inference is clearest when expressions are
written in terms of the empirical estimator of the crossentropy (Eqn. A2):
Ĥ(θ) ≡ − hlog q(X|θ)iX∈xN ,

(2)

where the angle brackets represent the empirical expectation (Eqn. A4). The marginal likelihood (i.e. evidence)
can be written [4]:
Z(xN ) ≡

Z

dθ $(θ) e−N Ĥ ,

A.

Application of thermodynamic identities

To extend the previously proposed correspondence,
we follow the standard prescriptions from statistical mechanics to compute thermodynamic potentials,
properties, and variables for the system [18, 20]. These
are shown in the lower half of Tab. I. The thermodynamic quantities depend on the particular realization
of the data X N . In the current context we are interested
in the expectation over this quenched disorder (i.e. data).
We define the disorder average with an overbar:
f (N ) ≡ h f (X N , θ0 ) iX,θ0 ,

(4)

(3)

Θ

which can be directly compared to the partition function in the canonical ensemble [1–6]. The model parameters θ are the variables that define the physical state
vector, the cross entropy Ĥ(θ) is the energy E(θ), the
prior $(θ) is the density of states ρ(θ).The data xN is
quenched disorder in the physical system. The sample size N is identified with the inverse temperature
β ≡ T −1 . (Choosing β ↔ N is only one of at least two
proposals for the identification of the temperature. See
App. A 2.) This assignment is natural in the following
sense: At small-sample-size N , many parameter values are consistent with the data, in analogy with the
large range of states θ occupied at high temperature T
in the canonical ensemble. The analogy between different quantities is summarized in Tab. I.

where X ∼ q(·|θ0 ) and θ0 ∼ $.

III.

RESULTS

We motivate interpretations of the thermodynamic
quantities by developing the thermodynamic potentials in the normal model and the large-sample-size
limit of singular models. The similarity between these
large-sample-size results and familiar results in statistical mechanics show that interpretations of the thermodynamic quantities of a statistical model can be deduced from the meaning of their physical counterparts.

3
Model

K-D-Free-particle
E0 +

K
2β

U

E0 +

K
2β

C

K
2

S

K
2

F

log

β
β0

K-D-Normal-prior
H0 +

K
2N

H0 +

K
2N

log

K-D-singular

N
N0

H0 +
H0 +
γ
2

K
2



1 − log

β
β0



K
2


1 − log

N
N0



γ
2N
γ
2N

log N + ...
+ ...

+ ...

− γ2

log N + ...

TABLE II. Thermodynamic-Bayesian correspondence. The thermodynamic quantities of a K-dimensional free particle with
ground-state energy E0 and thermal de-Broglie inverse temperature β0 , defined in the App. A 6 are compared to a Kdimensional normal model with a conjugate prior. Inspection reveals that the free particle is exactly equivalent to the normal
model, identifying the parameters as described in Tab. I. For the singular model, we supply only the leading order contributions
in the large N limit. The learning coefficient γ ≤ K. The special case of γ = K is a regular model.

A.
1.

B.

Models

Thermodynamic potentials
1.

Free particle and the normal model

We compare a free particle in K-dimensions confined
to a K-cube in statistical mechanics to a K-dimensional
normal model with K unknown means µ
~ and known
variance σ 2 in Bayesian inference. The true parameter
µ
~ 0 is drawn from a K-dimensional normal distribution
2
. It will be
(the prior $) with mean µ
~ $ and variance σ$
convenient to define a critical sample size:
2
N0 ≡ σ 2 /σ$
,

(5)

where the information content of the observations xN
is equal to the information content of the prior. (See
Appendix Sec. C 1.) In the current context, we will be
interested in the uninformative limit: N0 → 0.

2.

The normal model is representative of the large
sample-size-limit of a regular Bayesian model of dimension K (i.e. the Bernstein-von Mises theorem [21–
23]). For generality, we also study the large-sample-size
limit of a singular model of dimension K. Models are
singular when parameters are structurally unidentifiable
[24]:
for

A relation between the partition function and
Bayesian evidence has long been discussed [1–6]. The
free energy F represents the Bayesian model preference: the minus-log-evidence per observation, or message length per observation. In Tab II it is seen that
F breaks up into two parts. The first term is the code
length per observation using the optimal encoding, H0 ,
and the second term is the length of the code required
to encode the model parameters using the prior (per
observation) [25, 26]. The model that maximizes the
evidence and therefore minimizes F is selected in the
canonical approach to Bayesian model selection.
2.

Average energy

The thermodynamic prescription for computing the
average energy involves a derivative with respect to
temperature (Tab. I). In the context of statistics, we will
formally interpret this derivative using a finite difference definition, such that

Singular models

q(x|θ1 ) = q(x|θ2 )

Free energy

θ1 6= θ2 ,

(6)

where the unidentifiability cannot be removed by coordinate transformation. A regular model is the special
case where all parameters are identifiable, the parameter manifold is continuous, and the Fisher information
matrix (defined in Eqn. A6) is positive definite. Using
exact asymptotic results for singular models [24], the
thermodynamic quantities for each model and limit are
shown in Tab. II.

U (xN ) ≡ − log q(xi |x6=i )

i=1..N

,

(7)

where q(xi |x6=i ) ≡ Z(xN )/Z(x6=i ) is the Bayes-predictive
distribution. The RHS is a well-known statistical object:
the Leave-One-Out-Cross-Validation (LOOCV) estimator of model performance (See App. A 3). The statistical interpretation of average energy U is therefore the
minus-expected-predictive-performance of the model
(e.g. [27]).
As shown in the examples in Tab. II, the averaged
energy can be written as the sum of two contributions:
The first term H0 is the performance of the model if
the true parametrization was known and corresponds
to a ground state energy. The second term represents
the loss associated with predicting a new observation
X using estimated model parameters and corresponds
to the thermal energy. The loss term follows the typical behavior predicted by the equipartition theorem:
there is a half kB T of thermal energy per harmonic degree

4

3.

To study the predictive loss, it is natural to study the
statistical quantity corresponding to the heat capacity.
The heat capacity measures the rate of increase in thermal energy with temperature (C in Tab. I). The statistical analogue of the heat capacity, a learning capacity, is
a measure of the rate of increase in predictive performance with sample size. For the normal model:
(8)

as implied by the equipartition theorem. (See Tab. II.)
To consider how this analogy generalizes to a generic
statistical model, we use the large-sample-size limit
asymptotic expression for the Bayesian evidence from
Ref. [24] to compute the learning capacity. (See Tab. II.)
Like the normal model, the learning capacity for a singular model has the equipartition form but with an effective dimension:
C = 12 Keff ,

(9)

where Keff = γ is the learning coefficient defined by
Watanabe [24]. A regular model is a special case of this
expression where Keff = K, the dimension of the parameter manifold. We therefore conclude that equipartition theorem describes the universal properties of regular statistical models in the large-sample-size limit:
The learning capacity is half the number of degrees of
freedom.
.
4.

The Gibbs entropy

In physics, the Gibbs entropy generalizes the Boltzmann formula: S = log Ω where Ω is the number of
accessible states. We propose that the Gibbs entropy
has the analogous meaning in the context of Bayesian
statistics: The Gibbs entropy is the log-number of models consistent with the data. The finite-sample-size expression for the entropy is
S(xN ) ≡ N (U − F ),

(nats)

0
-2
-4
-6

Learning capacity

C = 12 K,

2

Entropy:

of freedom [18]. This is an important universal property
of regular models in the large-sample-size limit: Independent of the detailed structure of the model, there is
1
a universal predictive loss 2N
per degree of freedom in
the model. This universal predictive loss can be interpreted as the mechanism by which the Akaike Information Criterion (AIC) estimates the predictive performance [28, 29].

(10)

where Eqn. 7 provides an explicit expression for U . The
Gibbs entropy of a normal model is shown in Fig. 1.
Above the critical sample size N0 , the data is informative to the parameter values and therefore the number

10 -2

10 0

10 2

10 4

Relative sample size:
FIG. 1. Understanding Gibbs entropy. The Gibbs entropy
for the normal-model-with-prior is plotted as a function of
sample size. The Gibbs entropy can be understood heuristically as the log ratio of the model consistent with the data
to allowed models. At small sample size, the model structure determines the parametrization and therefore all models
allowed are consistent with the data and there is zero Gibbs
entropy. As the sample size grows beyond the critical sample
size N0 , fewer and fewer of the allowed models are consistent
with the data and the entropy decreases like − 21 K log N . The
non-positivity of the Gibbs entropy is a direct consequence of
the normalization of the prior, which forces the Gibbs entropy
to have a maximum value of zero. A prior determined by the
generalized principle of indifference avoids this non-physical
result.

of models consistent with the data is reduced. As a result the Gibbs entropy becomes increasingly negative
as sample size N grows.

C.

The principle of indifference

In statistical physics, the density of states is known
(i.e. measured) but in inference the selection of a prior
is often subjective. The construction of an objective
or uninformative prior is a long-standing problem in
Bayesian statistics. What insight does the proposed
correspondence provide for prior choice?
Prior construction since Bayes and Laplace has often attempted to apply a Principle of Indifference: All
mutually exclusive and exhaustive possibilities should be
assigned equal prior probability [30, 31]. One interpretation of this prescription is that it maximizes entropy [1, 32]. However, the principle of indifference
is difficult to interpret in the context of continuous
parameters, or across models of different dimension.
For example, are normal models with means µ and
µ + dµ mutually exclusive (distinguishable)? Even if
the mean were constrained to be an integer (µ ∈ Z)
to define mutually exclusive, the exhaustive condition
is also problematic. Exhaustive would correspond to
a uniform weighting over all integers. This vanishing
prior weight (1/∞) on the non-compact set Z results in
a paradoxical value for the evidence Z → 0 and the

5
rejection of the model irrespective of the data, as described in Sec. V B 1 (Lindley-Bartlett paradox).
D.

A generalized principle of indifference

To define mutually exclusive in a statistical context,
we look for natural analogues to this problem in statistical physics. A surprising result from the perspective of classical physics is that Nature makes no distinction between states with identical particles exchanged
(e.g. electrons) and counts only distinguishable states
(the Gibbs paradox). Following V. Balasubramanian
[4], we proposed that the concept of indistinguishability must be applied to objective Bayesian inference. We
take the mutually exclusive criteria in the principle of indifference to refer to distributions which are mutually
distinguishable at the experimental resolution available. We propose a generalized principle of indifference:
sets of indistinguishable models are each collectively
assigned the weight of a single distinguishable model.
To study the weighting of each model, we must prepare the data using a different procedure. We distribute
X N according to an assumed true parameter θ: X N ∼
q(·|θ), omitting the expectation over θ 1 . A generalized
principle of indifference states that the prior $ should
be chosen such that:
S(θ; N, $) ≈ const ∀ θ ∈ Θ,

(11)

at sample size N , where the Gibbs entropy is now a
function of θ. Eqn. 11 realizes a statistically principled
definition for the condition of equal model weighting
on mutually exclusive models.
The correspondence also offers a natural mechanism
for resolving statistical anomalies arising from the exhaustive condition in the principle of indifference that
gives rise to the Lindley-Bartlett paradox. In statistical
mechanics, the partition function Z is not normalized
by construction since the density of states ρ is a density
but not a probability density. Therefore, a natural solution to statistical anomalies arising from the exhaustive
condition is to re-interpret the objective inference prior
as a density of models. To specify a consistent density of
models between different parameter values and model
families, we replace the prior $(θ) with a model density w(θ) such that:
S(θ; N, w) ≈ 0,

(12)

assigning unit multiplicity to all parameters θ and
model families I. (Technical note: We avoid specifying
Eqns. 11 and 12 as equalities since the condition is typically not exactly realizable for all θ0 at finite sample

1

We must make the important distinction between the inference prior
$, used to perform inference, and the true prior, used to generate
the true parameters θ0 . This approach has a long and important
precedent: e.g. [24, 33].

size N . A precise formulation will be described elsewhere, but is analogous to the mini-max approach of
Kashyap where the largest violation of the GPI condition is minimized [33].) Eqn. 12, and the resulting prior
are reparametrization invariant (see App. A 5). The
prior w will be improper, but none-the-less the normalization is well defined. We shall refer to Eqn. 12 as the
Generalized Principle of Indifference which realizes both
the mutually-exclusive and exhaustive conditions using a principled statistical approach, regardless of the
nature of the parameter manifold. We will call the prior
w that satisfies Eqn. 12 the GPI prior.
IV.

APPLICATIONS

We have used the correspondence between thermodynamics and statistics to motivate the definition of the
learning capacity and the Gibbs entropy (and resultant
generalized principle of indifference). We now wish to
investigate the statistical properties of these definitions.
We will find that both these novel statistical objects provide new insight into statistics and learning.
A.

Learning Capacity

The applicability and failure of the equipartition theorem are well understood phenomena in physics. At
high or low temperature, degrees of freedom can become anharmonic, altering their contribution to the
heat capacity [18]. For instance, due to the discrete
structure of the quantum energy levels, degrees of freedom can freeze out at low temperature. (See Fig. 2A.)
Degrees of freedom can also become irrelevant at high
temperature. For instance, the position degrees of freedom of a gas do not contribute to the heat capacity
[34]. We see an analogous high-temperature freeze-out
mechanism in the context of inference.
In this section, we investigate the phenomenology of
the learning capacity in a series of simple examples. In
each case, we first compute F , then we compute the
learning capacity (as defined in Tab. I). In most of our
examples, both computations can be performed analytically.
• In Sec. IV A 2, we analyze the finite-sample-size
behavior of the learning capacity in the context
of the normal model with unknown mean and
variance. The dependence of the log-likelihood
on the variance is anharmonic and therefore there
are interesting finite-sample-size corrections to
equipartition.
• In Sec. IV A 3, we analyze a quantum-like freezeout phenomenon for a model on a discrete parameter manifold.
• In Sec. IV A 4, we analyze a problem where a discrete parameter manifold arise naturally: the sto-

6
(A)

Model Dimension:

4
7/2

Vibration (3x)
5/2

K=2

K=3

K=4

K=5

1£¹

2£¹
1£¹, ¾

3£¹
2£¹, ¾

4£¹
3£¹, ¾

5£¹
4£¹, ¾

A

Rotation (2x)

2

3/2
1

Translation (3x)

0
10

2

10

3

10

4

Learning capacity: C

Heat Capacity:

3

K=1

Temperature: T (K)
(B)

0.6

B

0.4

Learning capacity: C

Learning Capacity:

0.5

0.3
0.2
0.1
0.0
-0.1

10 -2

10 0

10 2

Relative Sample Size: N/N
Sample size: N

FIG. 2. The failure of equipartition. The behaviors of the
heat capacity and learning capacity are compared and related to the applicability or inapplicability of the Equipartition theorem in different regimes. Panel A: Low-temperature
freeze-out in a quantum system. The heat capacity is plotted as a function of temperature. Equipartition predicts that
the reduced heat capacity should be constant, equal to half
the degrees of freedom in the system. Plateaus can clearly
be observed at half-integer values, but the number of degrees of freedom is temperature dependent due to the discrete nature of quantum energy levels. At low temperature,
some degrees of freedom are frozen out since the first excited state is thermally inaccessible. This discrete topology of
the energy levels implies anharmonicity in the potential and
therefore failure of the equipartition theorem. Panel B: Hightemperature freeze-out in the Learning capacity. Analogous
to the statistical mechanics system, the statistical learning capacity transitions between half integer plateaus, reflecting a
temperature-dependent number of degrees of freedom. At
low sample size N (high temperature), the parameters are
completely specified by model constraints (the prior) and
therefore the parameters do not contribute to the learning capacity. At large sample size N , the parameters become data
dominated and therefore the learning capacity is predicted by
equipartition ( 12 K).

FIG. 3. Panel A: Learning capacity at finite sample size.
At large sample size, equipartition predicts the learning capacity of all models. At sample size N = 1 the learning
capacity diverges for models with unknown variance since
the mean and variance cannot be simultaneously estimated
from a single measurement. Panel B: Learning capacity on a
discrete manifold. The learning capacity of a normal model
with an unknown D-dimensional mean µ
~ ∈ ZD and variance
σ 2 = 15. For statistical uncertainty δµ  1, the learning capacity is predicted by equipartition since the discrete nature
of the parameter manifold cannot be statistically resolved.
For δµ  1, there is no statistical uncertainty in the parameter value (due to the discreteness of µ) and the degrees of
freedom freeze out, giving a learning capacity of zero.

tition for parameter values in the vicinity of the
singularity.
• In Sec. IV A 6, we analyze the learning capacity in
a non-regular but analytically-tractable model. In
this example the learning capacity is larger than
predicted by equipartition.

ichiometry of a Poisson processes.
1.

• In Sec. IV A 5, we analyze a singular model, the
exponential mixture model, which has previously
been identified as sloppy. We show that the learning capacity is smaller than predicted by equipar-

Normal model with an unknown mean

In many regular statistical models at finite sample
size, it is the model structure and not the data that constrain the parameter values. In these cases, structurally

7
constrained parameters will not contribute to the learning capacity. A simple and exactly tractable example
of this phenomenon has already been discussed: the
normal model with unknown mean and an informative
prior.
Model: We define a normal model on a D-dimensional
observation space with unknown mean and unknown
variance σ 2 . The likelihood function is:
q(~x|θ) ≡ (2πσ 2 )−D/2 exp[− 2σ1 2 (~x − µ
~ )2 ],

(13)

with θ ≡ (~
µ) and informative prior:
2 −D/2
$(θ) ≡ (2πσ$
)
exp[− 2σ12 (~
µ−µ
~ $ )2 ],
$

(14)

We now consider the informative limit where the critical sample size N0 (Eqn. 5) is finite.
Analysis: The learning capacity can be computed analytically:
C=

K
2(1+N0 /N )2 .

(15)

At large sample size, the learning capacity is equal to
the equipartition expression (Eqn. 9). At small sample size (high temperature), the prior determines the
parametrization, and therefore the parameter does not
contribute to the learning capacity and C → 0. (See
Fig. 2B.) This situation is roughly analogous to the heat
capacity of a gas. In the solid phase, the position degrees of freedom contribute to the heat capacity in the
canonical way whereas in the gas phase, the walls of
the box confine the atoms, the energy does not depend
on the position degrees of freedom, and these variables
no longer contribute to the heat capacity.
2.

Normal model with an unknown mean and variance

The learning capacity C typically deviates from
equipartition-like behavior in the small sample-size
(high temperature) limit in analogy to anharmonic effects in the heat capacity of metals near the melting
point [35, 36]. To demonstrate this phenomenon, we
analyze the normal model with unknown mean and
variance.
Model: We define a normal model on a D dimensional
observations space with unknown mean and unknown
variance σ 2 . The likelihood function is given by Eqn. 14
with θ ≡ (~
µ, σ) and impose an improper Jeffreys prior
$ = σ −D−1 . Although, the dependence of the information is harmonic in µ, it is non-harmonic in variance
σ2 .
Analysis: The learning capacity can be computed analytically:
h
2 (1) D(n−1)
C=D
( 2 ) + ...
(16)
2 1 − Dn ψ
i
2 3
2
(1) Dn
+ Dn
( 2 ) − D 4n ψ (2) ( D(n−1)
) , (17)
2 ψ
2

in terms of the polygamma functions ψ. The learning capacity is plotted as a function of sample size in
Fig. 3A. The learning capacity diverges at sample size
N = 1 since the mean and variance cannot be estimated from a single measurement and the divergence
of the learning capacity signals an infinite predictive
loss. The learning capacity of the normal model with
known mean and variance is representative of the behavior of many models: Typically, the learning capacity
can show significant differences with the equipartition
limit at very small sample size but rapidly converges to
the equipartition value as the sample size grows.
3.

Normal model with discrete mean.

An important exception to the generic behavior described in the previous paragraph occurs when the parameter manifold is discrete rather than continuous. In exact analogy to the freeze-out phenomenon in quantum
statistical mechanics, as the sample size increases (and
the temperature decreases) the discrete nature of the
parameter manifold (energy levels) becomes the dominant structure in the analysis and the system condenses
into a single distribution (ground state). We will consider a contrived but analytically tractable example: a
normal model with unknown discrete mean.
Model: The likelihood for the D-dimensional normal
model is defined in Eqn. 14. The parameters now include only the the mean: θ = (~
µ), with the mean constrained to have an integer values: µ
~ ∈ ZD . We assume
a flat improper Jeffreys prior: $ = σ −D .
Analysis: The learning capacity can be computed analytically and the expression is given in the Appendix
Sec. C 3. The learning capacity is plotted in Fig. 3B. To
discuss the phenomenology, it is useful to define a frequentist statistical resolution with respect to parameter
coordinate θi :
1p
δθi (N ) ≡ N − 2 [I −1 ]ii ,
(18)
in terms of the Fisher information matrix I (Eqn. A6),
which is a naturally covariant symmetric tensor on the
continuous parameter manifold. δθi (N ) is the width
of the posterior in the large-sample-size
limit. For the
√
normal model, δµ = σ/ N . For a regular model with
discrete parameters in the large sample size limit, the
learning capacity is:
(
1
K, ∆θi  δθi for all i
C= 2
(19)
0,
∆θi  δθi for all i
where ∆θi is the lattice spacing for parameter coordinate θi . The physical interpretation is clear: At large
sample size, the system condenses into a single state.
Therefore the corresponding degrees of freedom freeze
out, and no longer contribute to the learning capacity.
At small sample size, the discrete nature of the parameter manifold cannot be resolved, and the parameter

8

Stoichiometry of a Poissn process

In the previous example, we considered a somewhat
contrived example where the mean of a normal process was discrete, but many important statistical problems are naturally defined on discrete parameter manifolds (e.g. the dimension of a chi-square distribution,
the shape of gamma distribution, etc). For an explicit
example, we will analyze a problem that arises in the
context of our experimental work, the analysis of protein stoichiometry. The molecular complex is known to
be a multimer: a complex consisting of a well-defined
number of identical protein subunits. In our experiments, we measure stoichiometry by the fluorescence
intensity of the protein labels.
Model: The number of photons emitted per fluorophore
is well modeled by a Poisson process. We will assume
that the average intensity per fluorophore is known (i.e.
the rate λ is known). The stoichiometry, or number of
fluorophores, is m ∈ N, an unknown natural number.
Let xi be the binary observation of a photon (x = 1) or
no photon (x = 0) in a short interval length δt:
q(x|θ) ≡ (m λ δt)x (1 − m λ δt)1−x ,

(20)

where we will work in the limit as δt → 0 where the
parameter is θ = (m). We apply a flat prior $m = 1.
Analysis: The learning capacity can be computed analytically, as shown in Appendix Sec. C 7. The limiting cases can be understood intuitively in terms of
the results in the previous section (Eqn. 21). The lattice spacing
is ∆m = 1 and the statistical resolution is
p
δm = m/λt and the limiting learning capacity is:
(
C=

1
2,

0,

Relative posterior: (m|xN )

4.

(A)
1

Interval duration:
¸t = 10
¸t = 500

0
1

2

(21)

where the sample size dependence is represented as an
interval duration: t = N δt. For ∆m  δm, the stoichiometry m appears continuous since the posterior
on m spans multiple values of m (Fig. 4A) and therefore the learning capacity is predicted by equipartition
(Fig. 4B). For ∆m  δm, there is no statistical ambiguity in the parameter value m (Fig. 4A), the parameter
freezes out, leading to zero learning capacity (Fig. 4B).
The plot of the learning capacity (Fig. 4B) shows one
additional important feature that has already been discussed: At the smallest interval lengths t, the large
sample-size limit assumed in equipartition fails, leading to a second deviation from equipartition.

4

5

6

Stoichiometry: m

7

8

1/

2

0

10

100

1000

Interval duration: ¸t

FIG. 4. Panel A: Posterior for low-temperature freeze-out.
Low-temperature freeze-out occurs when there is no statistical ambiguity in the parameter value. For long interval durations (λt = 500) the posterior only weights a single parameter value (m = 6) whereas the manifold is effectively
continuous for intermediate interval durations (λt = 10) and
multiple parameter values as weighted. Panel B: Learning
capacity for low-temperature freeze-out. For long interval
durations (λt = 500), the stoichiometry m is frozen-out and
therefore the learning capacity is zero. For intermediate interval durations (λt = 10), equipartition applies and the learning capacity is one-half. At short intervals, the large-samplesize limit assumption is violated and the learning capacity
diverges from the limiting value.

5.

∆m  δm
∆m  δm

3

(B)
Learning Capacity:

manifold is effectively continuous. The learning capacity therefore assumes the equipartition value (provided that the sample size is large enough such that
the information is effectively harmonic, as discussed in
Sec. IV A 2).

Exponential mixture models.

The previous two examples demonstrated the
quantum-like freeze-out that results from discrete parameter manifolds. Our next example illustrates another small sample size (higher-temperature) freeze-out
phenomenon analogous to the loss of heat capacity
when a solid sublimes to form a gas. Here, this phenomenon arises as the result of model singularity: A
zero mode appears in the Fisher information matrix
corresponding to one (or more) parameter coordinates
becoming unidentifiable (Eqn. 6). To explore this phenomenon, we analyze the exponential mixture model
which has previously been identified as sloppy model
by Transtrum, Machta and coworkers using a criterion
defined by the distribution of the eigenvalues of the
Fisher information matrix [19].

9
Model: Consider a model for the lifetime of a mixed
population consisting of several different chemical
species I with different transition rates. Both the transition rates (kI ) and the relative abundance of the species
(pI ) are unknown. For an m species model, the likelihood function for the lifetime t is:
q(t|θ) ≡

m
X

pI kI e−kI t ,

(22)

I=1

with parameters:

θ≡

p1 ... pm
k1 ... km


,

(23)

subject to the constraint: I pI = 1 and we apply improper prior $(θ) = 1. The exponential mixture model
is singular since parameter kI is unidentifiable for pI =
0 and p1 is unidentifiable for k1 = k2 . (See Eqn. 6.) For
simplicity, we analyze the smallest model with a singularity (m = 2) to facilitate the numerical Bayesian
marginalization.

6.

Uniform distribution with unknown upper limit.

In the previous example, the non-regular model
showed reduced learning capacity at the singularity
but non-regular models can also have increased learning capacity as well. To illustrate this phenomenon,
consider a continuous version of the German Tank
problem, estimation of the support of a uniform distribution [37].
Model: Consider a uniform distribution X ∼ U(0, L)
with unknown end point L ≥ 0. The likelihood function is:
(
L−1 , 0 ≤ x ≤ L
q(x|θ) =
,
(26)
0,
otherwise

P

Analysis: We compute the learning capacity at two locations in parameter manifold, at the singularity (θS )
and far from it (θR ):

θS =

1 0
1 10




and

θR =

1
2

1
2

1 10


.

(24)

The learning capacity is computed numerically for N =
100 observations with distribution T N ∼ q(·|θ):
(
0.61, θ = θS
C(θ) =
.
(25)
1.5, θ = θR
Far from the singularity (θR ), the equipartition theorem
predicts the learning capacity (dim /2) whereas close to
the singularity (θS ), where the model is effectively described by only a single parameter (k1 ), the learning capacity reflects this smaller effective model dimension.
As expected, the exponential mixture model is predictively sloppy in the vicinity of the singular point, but
not elsewhere.
We expect that the behavior of the exponential mixture model is representative of many machine learning and systems biology problems. In these problems, the effective dimension Keff may be very much
smaller than the true dimension of model. In practice,
these models do not require exact structural unidentifiability (Eqn. 6) to show a reduced learning capacity. For instance, the reduced complexity is not only at
the singularity but in the vicinity of the singularity as
well. For models with structural unidentifiability, this
vicinity-of-singularity region will shrink with sample
size. For models without structural unidentifiability
(Eqn. 6) that are regular everywhere, equipartition will
hold at sufficiently large sample size.

with parameter θ ≡ (L) and improper prior $(L) =
L−1 .
Analysis: In this model, neither the first nor second
derivative of the cross entropy H(L; L0 ) exist at the true
parameter L0 and therefore the model is not regular. It
is straightforward to compute the learning capacity:
C = 1,

(27)

corresponding to an effective dimension of two, even
though the parameter manifold is one dimensional.
This result is exact and independent of sample size N .
(See Appendix Sec. C 6.)
B.

Generalized principle of indifference

The proposed generalized principle of indifference
has properties that rectify significant shortcomings
with other approaches to prior selection. We showcase
these properties with the analysis of a series of examples.
• In Sec. IV B 1, we compute the GPI prior for regular models in the large-sample-size limit. This
analysis reveals a connection between the GPI
prior and the Jeffreys prior.
• In Sec. IV B 2, we demonstrate an exact computation of the GPI prior for a number of nonharmonic models.
• In Sec. IV B 3, we demonstrate an exact computation of the GPI prior for a non-regular model.
• In Sec. IV B 4, we analyze the limiting behavior of
the GPI prior on a discrete parameter manifold.
GPI unifies two conflicting approaches, and interpolates between respective limits as a function
of sample size.
• In Sec. IV B 5, we analyze a problem where the
GPI prior cannot be computed exactly: the analysis of stoichiometry in a Poisson process.

10
1.

Approximate GPI prior for regular models

Model Dimension:
K=1

We will first explore the properties of the generalized
principle of indifference by computing the GPI prior in
the large-sample-size limit of a regular model. To define the GPI prior, it is first useful to define the scaledJeffreys prior:

N K/2
2π

I 1/2 ,

(28)

where I is the determinant of the Fisher information
matrix defined for a single sample (Eqn. A6), K is the
dimension of the continuous parameter manifold Θ.
The prior ρ is a density on parameter manifold with
the qualitative meaning of the inverse volume of indistinguishable models at sample size N . The GPI prior
is
w(θ; N ) ≈ ρ e−K ,

2.

Exact GPI prior for symmetric models

For simple models, symmetry and dimensional analysis often imply that w must still be proportional to the
Jeffreys prior even at small sample size. We compute
the exact GPI prior analytically for the normal model
with unknown mean and variance and the exponential model in the Appendix Sec. B 2. Both these models
have a log-likelihood that is anharmonic in the parameters and therefore are expected to have non-trivial hightemperature behavior. The calculation reveals that the
asymptotic form of the GPI prior (Eqn. 29) closely approximates the exact prior. In many models it is convenient to define the finite sample-size correction as an
effective complexity K that replaces the model dimension K in Eqn. 29:
w(θ; N ) ≈ ρ e−K,

(30)

K is plotted as a function of sample size (N ) for a number of different models in Fig. 5. On an empirical basis,
it is clear that Eqn. 29 is typically an excellent approximation for w even small to intermediate sample sizes
for many models.

3×

2×

2×

3×

Sample Size: N

(29)

where K = dim Θ, as shown in Appendix Sec. A 4.
In the large-sample-size limit, the parameter dependence of the GPI prior is identical to the Jeffreys prior,
which has enjoyed a long and successful history [38].
The Jeffreys prior was initially proposed because it was
reparametrization invariant [39]. More recently the
same prior has been motivated by numerous other arguments (e.g. [4, 40]). From the perspective of parameter inference, in the large-sample-size limit of a regular model, the GPI approach simply recapitulates a
widely-applied method rather than generating novel
algorithm.

K=3

Effective Complexity:

ρ(θ; N ) ≡

K=2
1×

FIG. 5. Effective complexity of models at finite sample size.
We computed the exact GPI prior for a series of models of
different dimension. At large sample size, the dimension
determines the effective complexity: K = dim Θ/2. At finite sample size there are significant corrections. The effective complexity divergences for the normal model (dashed
curves) with unknown mean and variance at N = 1.

3.

Exact GPI prior for a non-regular model

In this section we explore another key motivation to
the GPI approach: the analysis of non-regular models.
We return to the example of the uniform distribution
X ∼ U(0, L) with unknown end point L ≥ 0. In this
case, the Fisher information matrix (Eqn. A6) is not defined and so the Jeffreys prior approximation (Eqn. 29)
cannot be applied. The definition of w does not depend
on assuming a regular model and it is still straight forward to compute w (Appendix Sec. C 6):
w(L; N ) =

N
L



exp −1 − N log(1 + N −1 ) .

(31)

which has a scaling of sample size N corresponding to
an effective dimension of two, even though the parameter manifold is one dimensional.

4.

GPI prior for discrete parameter manifolds

For discrete parameter manifolds two competing
and well-established methods exist for choosing a
prior: (i) A literal interpretation of the principle of
indifference would seem to imply that all parameter
values are given equal weight. (ii) Alternatively, we
can consider the continuous parameter limit where the
prior can be chosen to give consistent results with the
Jeffreys prior. Both approaches have desirable properties in different analytical contexts [41]. GPI provides

11

5.

Stoichiometry of a Poissn process (revisited)

In each example discussed so far, it is possible to
compute the GPI prior exactly. In most applications
this approach is not tractable. Our analysis of regular
models suggests that Eqn. 29 is often an excellent approximation. If this does not suffice, a recursive algorithm (Appendix, Sec. B 4) is a practical refinement to
Eqn. 29. As an example of this approach, we return the
analysis of the stoichiometry of a Poissn process. (See
Sec. IV A 4.) In this case it is clear from the learning
capacity (Eqn. 21) that w must interpolate between the
discrete and continuous limits of Eqn. 32 as a function
of the stoichiometry m. We compute w numerically using the recursive algorithm (Appendix Sec. C 7).
We plot the w (GPI) and flat (PI) priors as a function
of the stoichiometry m in Fig. 6A and the Gibbs entropy
of the w and flat priors in Fig. 6B. The traditional interpretation of the principle of indifference leads to a
constant prior and non-constant Gibbs entropy. In contrast, GPI results in a non-constant prior and constant
Gibbs entropy. As is seen in Fig. 6B, the Gibbs entropy
of the flat prior rises for large stoichiometry because
models with stoichiometry m and m + 1 cannot be distinguished for the dimensionless interval length λt and
therefore the flat prior over weights these indistinguishable models by counting them independently. The GPI
prior compensates by weighting these large stoichiometry distributions less, resulting in constant Gibbs entropy. As the dimensionless interval (i.e. sample size)
increases, these distributions become increasingly distinguishable and the GPI increases the weight of these
models. For small stoichiometry where the models can
be statistically distinguished, the improper-flat prior
and w are identical (Fig. 6A).
C.

1.0

Prior:

0.8
0.6
0.4
0.2
0.0

(B)
(nats)

where ∆θi is the lattice spacing and the statistical resolution δθi is defined in Eqn. 18. The GPI prior for a
normal model with a discrete mean can be computed
exactly and is described in Appendix Sec. C 3.

GPI prior

(A)

Entropy:

an elegant resolution to this conflict: When the discrete
nature of the parameter manifold can be statistically resolved, the GPI prior prior assigns equal weight to all
discrete parameter values (i), whereas, if the discreteness of the space cannot be statistically resolved, the
large N limit gives rise to a Jeffreys prior (ii):
(
Q
ρ e−K i ∆θi , ∆θi  δθi
w=
(32)
1,
∆θi  δθi

3
2
1
0
5

10

15

Number of ﬂuorophores: m

20

FIG. 6. Complementary views on indifference: A flat prior
and the GPI prior offer two different ways to define the principle of indifference. Panel A: The GPI prior depends on
parameter and sample-size. The flat prior is constant with respect to changes in source number m, and sample size, while
the GPI prior (for Poisson stoichiometry problem) changes
with the parameter m, and responds to sample size. Panel
B: The GPI prior has (nearly) constant entropy. The average
entropy under the GPI prior is almost flat and zero everywhere, but the entropy of the flat prior is not constant. Some
models are entropically favored under the flat prior in violation of the generalized principle of indifference.

ters defined on non-compact manifolds. We consider
five competing models: three realizations of the normal model: known mean and variance, unknown mean
and known variance, and unknown mean and variance
and we also consider and exponential model with unknown rate and the uniform distribution model with
unknown end-point. (See Tab. III.) We will generate
datasets from all fives models and then perform inference on the model parameters and identity for each
dataset. We purposefully choose the true parameter
values and sample size (N = 20) such that cross entropies are not so large as to make prior choice irrelevant. (See Tab. III.)

Inference
1.

To demonstrate that the GPI prior automatically
leads to non-anomalous inference (i.e. free from infinite
normalization factors) and is also free from ad hoc parameters, we analyze simulated datasets for parame-

GPI approach

We have computed the GPI prior for each of the
proposed models. Inference on parameter values follows the standard Bayesian framework using the GPI

12
A

B

C

1

N
0.8

Model

0.6

N(µ, σ)
0.4

Exp(λ)
0.2

Model posterior probability

N(µ)

Generative model

Generative model

U(L)

Exp(λ)

N(µ, σ)

N(µ)

N

U(L)

Exp(λ)

N(µ, σ)

N(µ)

N

U(L)

Exp(λ)

N(µ, σ)

N(µ)

N

U(L)
0

Generative model

FIG. 7. Bayesian inference on model identity. The posterior of model identity (y axis) was computed for datasets generated
by each model (x axis). Panel A: GPI prior. For the simulated datasets, the generative model had the highest posterior probability as expected. Panel B: Normalized objective prior. The non-compactness of the parameter manifolds implies automatic
rejection of all the higher-dimensional models. In this case, since the model N is parameter-free, it has posterior probability of 1
for all datasets, regardless of the fit. Panel C: Revised informative prior. To avoid this undesirable anomaly, we tune the prior
parameter support to result in a reasonable posterior model probability. (See Tab. IV.) Inference is no longer objective, as the
posterior probabilities depend on how this tuning is performed. One representative plot of posterior probabilities to shown. In
general, inference cannot be expect to identify the generative model unless the KL divergence is so large as to make the prior
irrelevant.

Model

Parameter support

Generative parameters

θ

θ0
µ0 = 5, σ0 = 1

N
N(µ)
N(µ, σ)

µ∈R

µ0 = 6, σ0 = 1

µ ∈ R, σ ∈ R+

µ0 = 5, σ0 = 0.75

Exp(λ)

λ ∈ R+

λ0 = 2

U(L)

L ∈ R+

L0 = 10

TABLE III. Models for inference on simulated data. Five
data sets were generated, one for each model, using the generative parameters: X N ∼ q(·|θ0 ). Inference was performed on
the simulated data using the GPI prior w.

prior w. The GPI prior includes the model prior (Appendix Sec. A 7) and therefore the posterior probability
of model I is:
X
$(I|xN ) = ZI /
ZJ ,
(33)
J

where the model index J runs over the five competing models. The model posteriors for the five sets
of simulated data for a sample size of N = 20 are
shown in Fig. 7A. The results show a number of important characteristics of the GPI prior: (i) There is an
unambiguous Bayesian procedure for computing inference on both parameters and models. (ii) Inference on
both parameters and models leads to non-anomalous
results in which the generative distribution has nonzero posterior probability. (iii) For the normal models,
the higher-dimensional models have lower posterior
probability for the data generated by model N, even

though the generative distribution is realizable in N(µ)
and N(µ, σ). This shows that the GPI prior contains
an endogenous model selection mechanism favoring
model parsimony, and we will discuss this in detail in
Sec. V B 1.

2.

A canonical uninformative Bayesian approach

For contrast, we briefly describe a canonical Bayesian
approach to this analysis. We attempt to use the Jeffreys
prior for each model. A problem immediately presents
itself in the context of the Uniform model where the
Jeffreys prior is undefined. We must therefore deviate from our protocol and apply some other prior. We
set a flat prior on the parameter L motivated by the
principle of indifference. The priors for the four models with parameters cannot be normalized due to their
non-compact parameter manifolds. Parameter posteriors for each model can still be computed using an improper prior and the results are identical to the GPI approach, except for the uniform model where no Jeffreys
prior exists. If the Bayesian approach is interpreted literally, the model posterior for the parameter-free normal model is one, regardless of which distribution was
used to generate the data, due to the prior impropriety of the other models. (See Fig. 7B.) This is an undesirable outcome and this phenomenon is discussed in
more detail in Sec. V B 2.
A number of ad hoc modifications to the proposed
procedure are now possible to avoid this outcome. (i)
After having seen the data, a Bayesian will often reconsider the prior and localize it around the values favored
by the data. (See Fig. 7C.) This approach is formalized

13
A

V.

Sloppy
Regular

A.

Θ
Data generation:
q(X N |θ)

B

θ0

Θ

θ0
Parameter
Manifold

DISCUSSION

Regular

Inference:
(θ|X N )

1

XN

X

Θ
Observation
Manifold

1

Parameter
Manifold

FIG. 8. Panel A: Sloppiness is determined by parameter
manifold geometry and posterior width. Parameters are defined on a compact manifold Θ. In sloppy regions of parameter manifold, the parameters are model-structure dominated
(red posterior) whereas in regular regions of parameter manifold parameters are data dominated (green posterior). From
the perspective of the learning capacity, the model is effectively one dimensional in proximity to the red posterior and
two dimensional in proximity to the green posterior. Panel
B: Generalized principle of indifference. The posterior distribution $(θ0 |X N ) is shown schematically for two different
sample sizes N . The resolution increases with sample size as
the posterior shrinks. In the GPI prior (Eqn. 12), all parameter
values consistent with the posterior are assigned unit prior
weight collectively.

Learning capacity

One valuable feature of the proposed correspondence is the potential to gain new insights into statistical phenomenology using physical insights into the
thermodynamic properties of physical systems. Artificial Neural Networks (ANN) and systems-biology
models are two examples of systems with a large number of poorly-specified parameters that none-the-less
prove qualitatively predictive. This phenomena has
been discovered empirically and has been termed model
sloppiness [19, 47]. These models often have a logarithmic distribution of Fisher information matrix eigenvalues and this characteristic has been used as a definition of sloppiness [19]. But, this definition is unsatisfactory since it is not reparametrization invariant. It is
easy to construct counterexamples for this definition:
For instance, in a K-dimensional normal model where
the variance for each dimension is logarithmically distributed, the Fisher information eigenvalues are likewise logarithmically distributed, but the model nonethe-less behaves like a normal regular model from the
standpoint of prediction and statistical analyses.
The correspondence suggests a definition directly
written in terms of the predictive performance of the
model and the equipartition theorem. We propose
that predictive sloppiness be defined as models that have
a smaller learning capacity than estimated from the
model dimension:
C<

in variational or empirical Bayesian methods. In this
case, the prior is no longer determined a priori and this
double-use of data can lead to difficulties due to the potential for overfitting. (ii) A more rigorous approach is
to sub-divide the data: One subset is used to train the
prior to make it informative and the second set is used
to do inference using the canonical procedure. There
are two important disadvantages to this technique: An
ad hoc decisions must then be made about the size of
the data partitions. (This approach is essentially equivalent to eliciting a prior for the parameters from an expert.) A second disadvantage is that some information
is lost from inference on the model identity from the
data in the prior training subset. (iii) Alternatively, one
could use a non-cannocical approach for inference on
the model identity, like the use of the pseudo-Bayes
factor [42–46]. As we shall discuss below in Sec. V B 1,
this approach is consistent with the GPI approach in
the large-sample-size limit.
In summary, the GPI approach results in an unambiguous protocol for selecting the prior and then performing inference whereas the canonical Bayesian approach requires ad hoc modifications to lead to acceptable results.

1
2

dim Θ.

(34)

This definition (i) would exclude all regular models
in the large sample-size limit, (ii) is reparametrization
invariant and (iii) can be generalized to other nonBayesian frameworks by expressing the learning capacity in terms of the predictive performance.
The sloppiness phenomenon, or freeze out, is the
result of the model parameters being determined by
model structure rather than the data. To understand
the role of model structure in the freeze-out phenomenon, it is useful to write a qualitative expression
for the free energy F . We project the parameters into
a regular sector θR dimension KR and a singular sector θS dimension KS and assume the improper prior
$ = 1. We can then write:
1

N F ≈ N H0 − KR log (N IR )− 2 − KS log vS + ...,

(35)

where IR is the KR -root of the determinant Fisher
information matrix (Eqn. A6) projected onto the
regular sector.
The regular sector give rises to
an N -dependent KR -dimensional parameter volume
VR ≈ (N IR )−KR /2 whereas the singular sector gives rise
to a N -independent KS -dimensional parameter volume VS = vSKS . The resultant learning capacity is sloppy:
C = 12 KR < dim Θ,

(36)

14
with only the regular parameter coordinates contributing. This scenario is drawn schematically in Fig. 8A.
The posteriors for two parameter values are shown on
the parameter manifold: At a regular point (green),
KS = 0 and KR = 2 and all parameter coordinates are
regular and data dominated. At the sloppy point (red),
the manifold is not rigorously singular but KS is effectively 1 since the manifold constraints determine the
parameter value in the vertical coordinate direction.
In summary, it is parameters inference dominated by
the model structure rather than data that in each case
give rise to the anomalously small learning capacity
and the qualitative phenomenon of anomalously predictive models. The learning capacity has the potential
to offer new insights into the mechanism of learning in
more complex systems, including ANNs. Our preliminary investigations suggest some training algorithms
may map to physical systems with well-understood
thermodynamic properties. The detailed physical understanding of the complex phenomenology of physical systems, including phase transitions, renormalization, etc., have great promise for increasing our understanding of the fundamental mechanisms of learning.

1.

Model selection

The normalization of the GPI prior has significant consequences for inference on model identity
(i.e. model selection). Returning to the regular model,
it is straight forward to apply the Laplace approximation to compute the minus-log evidence using the GPI
prior:
− log Z(xN ; w) ≈ − log q(xN |θ̂) + K,

where K = dim Θ. The scaled-Jeffreys prior cancels the
Occam factor from the integration. The two remaining contributions each have clear qualitative interpretations: the MLE estimate of the information (− log q)
and a penalty for model complexity (K). Eqn. 37 is already well known as the Akaike Information Criterion
(AIC)2 :
− log Z ≈ AIC(xN ).

The generalized principle of indifference

We argue that a natural approach to objective
Bayesian inference is to choose a prior such that the
number of indistinguishable distributions is one for all
parameter values. This generalized principle of indifference is simply written S = 0. (See Eqn. 12.) Schematically, this procedure assigns equal prior weighting to
all models that can be distinguished at finite sample
size N . As the sample size increases, the prior must
be modified to accommodate the increased resolution,
i.e. (δθi )−1 , due to shrinking of the posterior support.
It is important to stress that GPI gives rise to a samplesize-dependent prior and therefore this inference is not
Bayesian in a classical sense: (i) It violates Lindley’s
dictum: today’s posterior is tomorrow’s prior. (ii) Furthermore, the evidence and prior are no longer interpretable as probabilities but rather statistical weightings. On-the-other-hand, the method codes parameter
uncertainty in terms of a posterior probability distribution and facilitates Bayesian parameter and model
averaging. Therefore, we would argue the approach
maintains all of the attractive features of the Bayesian
framework while avoiding problematic aspects. At
small sample size or in singular models, the GPI prior
must be computed explicitly. (See Fig. 8B.) For a regular model in the large-sample-size limit, no calculation
is required and GPI prior is equal to the scaled-Jeffreys
prior (Eqns. 28.)

(38)

Information-based inference is performed by selecting
the model which minimizes AIC, maximizing the estimated predictive performance. The reason why AIC
and GPI Bayesian inference are equivalent is most easily understood by rewriting Eqn. 12:
− N F ≈ −N U ,

B.

(37)

(39)

which in statistical language corresponds to using a
prior that makes the log partition function (LHS) an
unbiased estimator of the log predictive performance
(RHS). Since the Akaike Information Criterion (AIC)
is an unbiased estimator of RHS at large sample size
N , the generalized principle of indifference encodes
an AIC-like model selection [48] and an informationbased (AIC) realization of Occam’s razor: parsimony increases predictivity [29]. The log-predictive performance
(RHS Eqn. 39) has been advocated in the context of
Bayesian model selection through the use of pseudoBayes factors by Gelman and coworkers [42–46].
2.

Lindley-Bartlett paradox

In contrast to the unambiguous inference generated by the GPI approach, the canonical uninformative
Bayesian approach leads to a number of difficulties and
ambiguities, as we discovered in Sec. IV C. These difficulties arise due to the normalization of the prior. We
can compute the total number of distinguishable distributions defined by the GPI prior in model I at sample
size N by integrating the GPI prior over the parameter
manifold:
Z
MI (N ) ≡ dθ w(θ; N ).
(40)
Θ

2

where AIC is defined in nats (rather than the more common deminat expression which is twice Eqn. 37)

15
For non-compact manifolds, MI (N ) may diverge, but
this is of no significance. A more detailed discussion of
the Bayesian meaning of this procedure is provided in
the Appendix Sec. A 7.
To explore the significance of this normalization,
we will compute the minus-log evidence of a regular
model in the large-sample-size limit with a normalized Jeffreys prior. As described above, this prior is
equivalent to w normalized by the distribution number
(Eqn. 40):
$J (θ) = M −1 w(θ).

(41)

Using the Laplace approximation, it is straightforward
to compute the minus-log-evidence:
− log Z(xN ; $J ) = − log q(xN |θ̂) + log M + O(N 0 ),

(42)

where where the first term (order N ) has an interpretation of goodness-of-fit, the second term (order log N )
is the minus-log Occam factor or log-number of distinguishable distributions (Eqn. 40). The corresponding
GPI expression is Eqn. 37.
The difficulty with using evidence in Eqn. 42 as a
measure of statistical support is clearest in the context
of a simple example: the normal model. Consider the
analysis of a normal model with N observations xN
and known variance σ 2 . Consider two competing models: mean is zero (the null hypothesis) and µ 6= 0 (the
alternative hypothesis). We must define an acceptable
range of values for µ. Assume a flat proper prior on an
interval length L. The log number of distinguishable
distributions is approximately:
log M = log

L
δµ

1
2π

 12

+ ...

(43)
1

where the error in the mean is δµ ≡ σ/N 2 . The condition that the evidence is greater for the µ 6= 0 model
than for the µ = 0 model is:
( 1
22 ,
GPI
µ̂ ≥ δµ
.
(44)
1
2
(2 log M ) , normalized
The expression for GPI is independent of the interval
length L whereas the normalized expression still retains a dependence on L. In fact, in the limit that the
prior is uninformative (L → ∞), no finite observed
mean is large enough to support the µ 6= 0 model
for model selection with a normalized prior. We have
described just such an example in Sec. IV C. This automatic rejection of high-dimensional models in the
context of uninformative priors is called the LindleyBartlett paradox [49, 50]). The use of the GPI prior circumvents this anomalous result.
3.

One such exception is shown in Fig. 5 which reveals
that the normal model with unknown mean and variance has a divergent effective complexity at a sample
size of N = 1. The effect of this divergence is to give
these models zero statistical weight. Although this may
initially appear problematic, it is an important feature
of the generalized principle of indifference. A mean
and variance cannot be estimated from a single observation and as a result the model parameter posterior
would be improper and the predictive loss would be
infinite. Therefore the generalized principle of indifference automatically gives this model zero statistical
weight (Z = 0). The inability of other approaches to automatically handle posterior impropriety is recognized
as a significant shortcoming [38].

Posterior impropriety

The use of GPI prior often, but not always, gives
non-zero evidence for all models under consideration.

C.

Comparison with existing approaches and novel
features

The generalized principle of indifference subsumes a
patchwork of conflicting methods for prior and model
selection, resolving many conflicting approaches and
generating a single, generally-applicable and selfconsistent framework. The GPI approach subsumes
the following approaches: (i) For discrete parameter
manifolds in the large sample size limit, the GPI gives
equal weight to all mutually exclusive models, consistent with the original formulation of the principle
of indifference by Bayes and Laplace [30, 31]. (See
Eqn. 32.) (ii) In the large-sample-size limit, GPI generates a GPI prior proportional to the well-known Jeffreys prior. In this sense, the approach is closely related to the reference prior approach of Bernardo and
Berger [40, 51]. (iii) With respect to model selection
(inference on model identity), the GPI evidence behaves like pseudo-Bayes factors (or AIC) and therefore circumvents the Lindley-Bartlett paradox. (See
Sec. V B 1.) To date, the pseudo-Bayes approach has always been un-Bayesian in the sense that the pseudoBayes method consists of the ad hoc combination of a
canonical Bayesian prior for inference on parameters
but a cross-validation-based weighting for inference on
models. The GPI provides a self-consistent equivalent
approach to inference on both parameters and models.
The GPI addresses a number of problems with
existing approaches to objective Bayesian inference.
(iv) Lindley-Bartlett paradox: As already discussed
above, an important shortcoming with existing objective Bayesian approaches relates to the compactness
of the parameter manifold and the automatic rejection of higher-dimensional models in model selection
(the Bartlett-Lindley paradox [49, 50]). More generally, the evidence of the canonical objective Bayesian
approach depends on ad hoc modeling decisions, like
the range of allowed parameter values. The GPI-Bayes
evidence circumvents these anomalies by generating a
consistent distribution density w over competing models. As a result the GPI evidence is independent of ad

16
hoc modeling decisions. (See Sec. V B 2.) (v) Unification of statistical paradigms: The absence of the LindleyBartlett paradox implies coherent inference between
paradigms [52, 53] and therefore the generalized principle of indifference naturally unifies objective Bayesian
inference with information-based inference. (vi) Prior
and posterior impropriety: Another important flaw identified in other objective Bayesian approaches is the inability to handle impropriety. In many cases where parameters are defined on non-compact manifolds, the
prior (and sometimes the posterior) cannot be normalized. The redefinition of the prior as a density of models introduces a well-defined and consistent method for
defining prior normalization, regardless of the global
structure of the manifold. Furthermore, the approach
automatically assigns zero statistical weight to models
that suffer from posterior impropriety. (See Sec. V B 2.)
(vii) Discrete parameter manifolds: The GPI-Bayes approach also unifies two well-established approaches to
defining objective prior on discrete manifolds: equal
weight versus Jeffreys prior. GPI Bayes interpolates between these two limits as a function of sample size. (See
Eqn. 32.) (viii) Singularity and sloppiness: Finally, the
GPI-Bayes approach does not assume model regularity.
It treats singularity and the sloppiness phenomenon in
a natural way. (See Sec. IV B 2.)

D.

Conclusion

Nature reveals an elegant formulation of statistics in
the thermal properties of physical systems. Measurements of the heat capacity, compressibility or susceptibility reveal unambiguously how Nature enumerates
states and defines entropy. These physical insights provide clues to the definition of novel statistical quantities
and the resolutions of ambiguities in the formulation of
objective Bayesian statistics. We have refined a previously proposed correspondence between the Bayesian
marginal likelihood and the partition function of statistical physics. We demonstrate a novel and substantive mapping between the average energy, heat capacity, entropy and other statistical quantities. The
newly-defined learning capacity is a natural quantity
for characterizing and understanding learning algorithms and generates new insight into the Akaike Information Criterion (AIC) and model sloppiness through
a correspondence with the equipartition theorem and
the freeze-out phenomenon, respectively. Finally, we
use the Gibbs entropy to define a generalized principle of indifference and an objective Bayesian weighting
prior with the property that all distributions have equal
prior weight. This approach subsumes many seemingly inconsistent and disparate methods into a single,
coherent statistical approach.
Acknowledgements: PAW and CHL acknowledge
helpful discussions with M. Linden, J. Kinney,
D. Mayo, C. Heilig, M. Abbott, B. Machta and

M. Transtrum. This work was support by NSF grants
NSF-PHY-084845 and NSF-MCB-1151043-CAREER.

17
Appendix A: Supplemental results
1.

Definitions of information, cross entropy, Fisher information matrix

The Shannon Information is defined:
h(x|θ) ≡ − log q(x|θ).

(A1)

Let X be distributed with a true distribution with parameter θ0 : X ∼ q(·|θ0 ). The cross entropy is defined:
H(θ; θ0 ) ≡ h(X|θ),

(A2)

H0 (θ0 ) ≡ H(θ0 ; θ0 ).

(A3)

and which has a minimum at the true entropy:

The empirical estimator of the cross entropy is defined:
Ĥ(θ) ≡ N −1

N
X
i=1

h(xi |θ),

(A4)

which scales like N 0 in spite of the prefactor. The KL-Divergence:
DKL (θ0 ||θ) = H(θ; θ0 ) − H0 (θ0 ),
is the natural distance-like measure on the parameter manifold. The Fisher information matrix is defined:


Iij = ∂θ∂ i ∂θ∂ j H(θ; θ0 ) θ=θ0 ,

(A5)

(A6)

which is a rank two covariant tensor known as the Fisher-Rao metric.
2.

An alternate correspondence

In establishing the correspondence between inference and statistical mechanics, we identify the partition function Z as the marginal likelihood and N ↔ β in agreement with V. Balasubramanian [4]. This is not the only
choice. For instance Watanabe [24] instead chooses to define the inverse-temperature β so that the likelihood is
given by q β (X N |θ), that is raised to an arbitrary power β. This identification has two advantages: i.) It seems to
be more closely related to the physical temperature, which can be varied independently with the strength of the
quenched disorder ii.) It allows one to interpolate between a Bayesian posterior (given by β = 1) and the point
estimates of the MLE’s (given by β = ∞). This temperature has also been applied in tempering schemes in MCMC
methods, and simulated annealing—increasing the temperature promotes a better exploration of the sample space
(chain-mixing) that can be used to better sample multimodal distributions, or find the minima in a rough function.
On the other hand, there are two disadvantages of a power β relative to N ↔ β which we believe outweigh the
advantages: i.) First, it is not a preexisting statistical parameter within the Bayesian framework. ii.) Second, the
internal energy under this other choice of β is not the predictive performance U . Consequently, the principle of
indifference which results from a likelihood-power β does not induce the Akaike weights as the model averaging
procedure. Instead U = H0 , which does not encode a realization of Occam’s razor.
Thermodynamic expressions using both definitions may give somewhat complementary information. Which
is useful will depend on the context. We do not believe that statistical mechanics prescribes a uniquely-correct
procedure for objective Bayesian inference. It is the reproduction of a principled model selection criteria, AIC with
its proven asymptotic efficiency [54] that justifies the proposed correspondence in the context of model selection.
3.

Finite difference is equivalent to cross validation

The log-predictive distribution can be written as a finite difference
log q(Xi |X 6=i ) = log Z(X N ) − log Z(X 6=i ),

(A7)

18
We can interpret the log q(Xi |X 6=i ) as a finite difference estimate of the the sample size derivative of the free energy.
We take the mean over all permutations of the data so that this estimate is symmetric with respect to all data points.
Under expectation, analytically continuing sample size, the LOOCV relationship to the internal energy is clear:
hlog q(Xi |X 6=i )i ≈

∂
hlog Z(X N )i + O(N −1 ).
∂N

(A8)

This identity is crucial in establishing the thermodynamic interpretation in terms of predictive performance
4.

Jeffreys prior is proportional to GPI prior in the large-sample-size limit

In the large-sample-size limit, the partition function can be evaluated using the Lapalce (saddle-point) approximation and the resulting prior is proportional to the Jeffreys prior. The integral is evaluated by expanding around the minimum of ĤX (θ), the maximum likelihood estimator: θ̂X . The partitition function Z(X N ) =
R
dθ $(θ) exp[−N ĤX (θ)], becomes
Θ
Z(X N ) ≈ e−N ĤX (θ̂X )



2π
N (det I)1/K

K/2
$(θX )

By the standard χ2K representation of the overfitting error, hĤ(θ̂X )iX = H0 −
becomes
F (θ0 , $, N ) = H0 −

(A9)
K
2N .

Therefore the disorder average

K
K
2π
1
−
log
−
log $(θ0 ) + O(N −2 )
2N
2N
N
N (det I)1/K

(A10)

We can then calculate the Gibbs entropy N 2 ∂N F ,
S(θ0 , $, N ) =

2π
K
log
+ K + log $(θ0 ) + O(N −1 )
2
N (det I)1/K

(A11)

If we enforce the generalized principle of indifference, ignoring higher powers of N −1 ,
0 = S(θ0 , w, N )

(A12)

and substituting the w for $ in the entropy expression then gives us the conditon
w(θ0 ) = (det I)1/2



N
2π

K/2

e−K .

(A13)

Thus the generalized principle of indifference is satisfied by the Jeffries prior in the large-sample-size limit. The
constant weighting factor is important in model selection as e−K expresses the Akaike weighting.
This constant factor shows another important characteristic of the GPI prior: it has sample-size dependence. This
sample size dependence will in general break the de-Finetti likelihood principle: that the prior should not depend
on the nature of the data-generating procedure (including the sample size). The departure from the likelihood
principle is the origin of the departure from the conventional Bayesian model selection behavior.
5.

Reparametrization invariance of thermodynamic functions and the GPI prior

Reparametrization invariance of the thermodynamic quantities follows from there being derived from the partition function which is also reparametrization invariant. The partition function under an invertible coordinate
transformation becomes
Z
Z(X N ) = dφ q(X N |φ)$ (θ(φ)) J(φ).
(A14)
At the same time the density transforms so that $ (θ(φ)) J(φ) → $0 (φ) where J is the determinant of the Jacobian. Notably, if $ (θ) satisfies the GPI, then the transformed density $ (θ(φ)) J(φ) → $0 (φ) results in the same
partition function and Gibbs entropy and therefore still satisfies the GPI in this new coordinate system.

19
6.

Effective temperature of confinement

To calculate the free energy Fof a free particle confined to a volume V = L3 , we calculate the partition function
by integrating over available phase space:
Z K K
d p d x −βH(p,x)
Z(β) =
e
(A15)
(2π~)K
Z
K 
K/2
−βp2
mL2
e−βE0 LK
dp e 2m
=
e−βE0 .
(A16)
=
(2π~)K
2π~2 β
The Free energy is then
F(β) = E0 +

K
mL2
K
β0
log
= E0 +
log
2β
2π~2 β
2β
β

(A17)

where we have made the identifications
β0 =

mL2
2π~2

and K = 3.

(A18)

β0 can be interpreted as the inverse of the (typically negligibly small) temperature at which the thermal de Broglie
wavelength of the confined particle is on the order of the width of the confining box.
7.

A Bayesian re-interpretation

The replacement of the prior (a probability density) with an unnormalized density of states may make a Bayesian
reader uncomfortable since the evidence (Z) no longer has the meaning of a probability. But there is a natural Bayesian interpretation in terms of the a priori model probability. Typically, when models are compared in a
Bayesian context, all mutually exclusive models are assigned equal a priori probabilities (i.e. the principle of indifference). But, we have now proposed a new concept of model enumeration by introducing a density of models.
We can compute the total number of distinguishable distributions in model I at sample size N by integrating the
GPI prior (density of states) over the parameter manifold:
Z
NI (N ) ≡ dθ wI (θ; N ).
(A19)
Θ

Since models I and J contain different numbers of distinguishable distributions, we reason that the principle of
indifference should be interpreted to apply at the distinguishable distribution level rather than the model level.
Therefore the a prior model probabilities should be:
P
$I ≡ NI / I NI .
(A20)
and the proper parameter prior is
$(θ|I) ≡ wI (θ; N )/NI .

(A21)

Inference with the improper GPI prior is equivalent to assuming proper prior $I on models and proper prior
$(θ|I) on parameters. The numerator in RHS of Eqn. A20 will cancel the denominator in the RHS of Eqn. A21
when the model posterior is computed and the normalization NI divides out of parameter posterior distributions.
Appendix B: Methods
1.

Computation of learning capacity

To compute the learning capacity, we will use the definition from Tab. I:
2
C(θ; N, $) = N 2 ∂N
log Z(θ; N, $),

where X ∼ q(·|θ).

(B1)

20
2.

Direct computation of GPI prior

We will use the discrete difference definition of the entropy (Eqns. 7 and 10) to enforce the generalized principle
of indifference (Eqn. 12). The relation for the GPI prior can be written:
(N + 1) log Z(θ; N, w) = N log Z(θ; N + 1, w),

(B2)

in terms of the partition function. We will use Eqn. B2 explicitly to solve for the GPI prior w. For the models
we work analytically, we will be be able to use the asymptotic form of w (Eqn. 29) to define an effective model
dimension K (Eqn. 30). The general strategy will be:
1. Use symmetry and dimensional analysis to deduce the scaling of w with respect to the parameters θ.
2. Compute log Z(X N ; w) and re-express in terms of canonical random variables.
3. Compute log Z(X N ; w).
4. Solve for the unknown normalization c of w using GPI (Eqn. B2).
3.

Computation of the free energy using a sufficient statistic

It is often convenient to work in terms of sufficient statistics because (i) all the data dependence of the posterior
enters through the sufficient statistic and (ii) the statistics have well known statistical distributions that significantly
simplify many calculations. We define a sufficient statistic t = T (X N ) such that
Pr(θ|X N ) = Pr(θ|t),

(B3)

or all the information about the parameters is encoded in t. We can therefore write:
q(X N |θ) = q(X N |t) q(t|θ),

(B4)

and we can define a Statistic Shanon entropy:
Ht (θ) = −log q(t|θ).

(B5)

In terms of the sufficient statistic, the partition function factors:
Z(X N ; $) = q(X N |t) z(t; N, $),

(B6)

where the statistic partition function is
z(t; N, $) ≡

Z
dθ $(θ) q(t|θ).

(B7)

Θ

The expected free energy can be written:
F (θ; N, $) = −N −1 log z(t; N, $) + H0 (θ) − N −1 Ht (θ),

(B8)

where H0 is the entropy.
4.

Computation of the GPI prior using a recursive approximation

The Gibbs entropy has the property that it is linear in the prior so that the following holds:
S(θ0 , N, eα $) = α + S(θ0 , N, $)

(B9)

If the prior and entropy are flat, then setting α = −S(θ0 , N, $) will result in S(θ0 , N, eα $) = 0; the w-prior
condition. This suggests the following simple recursive scheme for a successive approximation for the w-prior:
1: procedure R ECURSIVE W($)
2:
repeat

21
$(θ) ← $(θ)e−S(θ,$,N )
until S(θ; $, N ) ≈ 0
end procedure
To the extent the entropy is slowly varying and only locally dependent on the prior, this algorithm will very
quickly converge to an exact w-prior. However, effects due to manifold boundaries and model singularities may
create artifacts that lead to unstable updates. Empirical evidence suggests that the algorithm should be terminated
before the exact GPI prior condition is met. Typically very few iterations are required. In the Poisson stoichiometry
problem, w for t = 100 and t = 500 were calculated with only a single iteration. At smaller sample-sizes, more
iterations are required.
3:
4:
5:

Appendix C: Details of applications
1.

Normal model with unknown mean and informative prior

The likelihood for the normal model is defined by Eqn. 14 with parameters θ ≡ (~
µ) for support µ ∈ RK for a
2
normal model with unknown mean and known variance σ . In this example, we assume a conjugate prior:
2 −K/2
$(θ) = (2πσ$
)
exp[− 2σ12 (~
µ−µ
~ $ )2 ],

(C1)

$

2
. The partition function is computed by completing the
where we introduce the critical sample size N0 ≡ σ 2 /σ$
N
square in the exponential. If X ∼ q(·|θ) and θ ∼ $, the log partition function can be expressed in terms of three
independent chi-squared random variables:

σ −2

N
X
~i − µ
(X
~ˆX )2 ∼ χ2K(N −1) ,

(C2)

i=1

σ −2 N (~
µ−µ
~ˆX )2 ∼ χ2K ,
σ −2 N0 (~
µ−µ
~ $ )2 ∼ χ2K .

(C3)
(C4)

The log partition function is therefore distributed:
2
log Z(X N ; $) ∼ − KN
2 log 2πσ −

K
2

log

N +N0
N0

− 12 χ2K(N −1) −

1 N0 N
−1 2
χK
2 N +N0 (N

0

+ N0−1 χ2K ),

(C5)

where χ2j is a chi-squared random variable dimension j and the expect-log partition function is
log Z(N, $) = −N H0 −

K
2

log

N +N0
N0 ,

(C6)

where H0 is the entropy and the free energy is:
F (N, $) = H0 +

K
2N

log

N +N0
N0 .

(C7)

The other results in the Tab. II are generated by apply the definitions of the correspondence in Tab. I.
2.

Normal model with unknown mean

The likelihood for the normal model is defined by Eqn. 14 with parameters θ ≡ (~
µ) for support µ
~ ∈ RD for a
2
normal model with unknown mean and known variance σ . The cross entropy is


1
2
H(θ; θ0 ) ≡ D
µ−µ
~ 0 )2 ,
(C8)
2 log 2πσ + 1 + 2σ 2 (~
where the true distribution is X ∼ q(~x|θ0 ) and the determinant of the Fisher information matrix is:
det I = σ −2D .

(C9)

N
D/2
ρ = ( 2πσ
.
2)

(C10)

The scaled Jeffreys prior (Eqn. 28) is therefore:

22
We will assume w matches the asymptotic form:
w = cσ −D ,

(C11)

and solve for the unknown constant c(N, D). The partition function is
log Z(X N ; w) ∼ log c −

DN
2

log 2πσ 2 +

D
2

log

2π
N

− 12 χ2D(N −1)

(C12)

where χ2D(N −1) is a D(N − 1)-dimensional chi-squared random variable. The expected log partition function is:
log Z(θ; N, w) = −N H0 (θ) + log c +

D
2



log

2π
N


+1 ,

(C13)

where H0 is the true entropy. The learning capacity is:
C(θ; N ) =

D
2,

(C14)

where D is both the dimension of mean parameter and the model. The unknown normalization of w is:


N
D
−1
log c = D
)
2 log 2π − 2 1 + N log(1 + N
which can be re-written as an effective dimension:


−1
K= D
) ,
2 1 + N log(1 + N

(C15)

(C16)

to define the GPI prior w using Eqn. 30.
3.

Normal model with unknown discrete mean

The likelihood for the normal model is defined by Eqn. 14 with parameters θ ≡ (~
µ) for support µ
~ ∈ ZD for
2
a normal model with unknown mean and known variance σ . We use Eqn. B8 to treat the problem in terms of
sufficient statitics. The statistic partition function breaks up by dimension: For each dimension with flat prior
P
PN
$(µ) = m δ(µ − m), and sufficient statistic t = N −1 i Xi the statistic partition function becomes the sum over
discrete prior values:
∞
X

z(t; N, $) =


q(t|m) =

m=−∞

N
2π

1/2 X
∞

e−N (t−m)

2

(C17)

m=−∞

∞


X
−2π 2
2
= ϑ t; r = e N
=
rm e2πt

(C18)

m=−∞

Where ϑ is the Jacobi theta function with nome r. We can use the Jacobi triple product formula to write down a log
partition function
log z(t; N, $) =

∞
X
m=1



log(1 − r2m ) + log 1 + r2m−1 e−i2πt + log 1 + r2m−1 ei2πt

(C19)

We are now able to take the disorder average analytically. Assume (without loss of generality) that m0 = 0, then,
because |r| < 1, we can safely expand the logarithm, and
∞ X
∞
X

(−1)k (2m−1)k
log 1 + r2m−1 e−i2πt = −
r
E e−i2πkt
k
t|m0
t|m0
m=1
m=1

E

∞
X

(C20)

k=1

The expectation is just φN (2πk), the characteristic function of the central normal distribution with variance σ 2 =
1/N . Specifically φN (2πk) = e−
−

2π 2 k2
n

= qk

2

2
∞ X
∞
∞
X
(−1)k (2m−1)k+k2 X (−1)k rk +k
r
=
k
k r2k − 1
m=1

k=1

k=1

(C21)

23
This series is convergent, but converges slowly for very large N . We therefore must also develop a series for when
N  1. We can use the Poisson resummation formula to convert the partition function into a sum over reciprocal
space. First we have to subtract off the singularity at zero, by adding a piece to the summand that can be explicitly
summed. Then we can extend this function to both positive and negative integers:
∞
X

∞
2 2
 1X
k
cos(πk)
− 2πNk
log 1 + r2m−1 e−i2πt =
2k  e
2
2π
2
k
t|m0
sinh N
m=1

E

(C22)

k=1

∞
n
n X cos(πk)
=
+ 2
48 4π
k2

2π 2 k2
2π 2 k
 e− N − 1
2π 2 k
n sinh N

k=1

∞
1
π2
n X cos(πk)
n
+ −
+ 2
=
48 4 12n 8π
k2
k=−∞

!
(C23)

!
2 2
2π 2 k
− 2πNk
−1
2 e
n sinh 2πN k

(C24)

The sum can now be represented as the sum of the Fourier transform of the summand. At large n, even the first
term is exponentially small, and the whole sum can be ignored, leaving:
(P
2
∞
∞ (−1)k r k +k
X

for all n
2m−1 −i2πt
= n k=11 k π2r2k −1
E
log 1 + r
e
(C25)
t|m0
n > 102
m=1
48 + 4 − 12n
Similarly we have for the Euler function piece of the Jacobi-theta triple product
(P
∞
∞ 1 r 2k
X
for all n
2m
k=1 k 1−r 2k

log(1 − r ) =
π2
1
n
n
n > 50
− 24 + 6n + 2 log 2π
m=1

(C26)

The rational tail from these two contributions in the asymptotic expansion cancel exactly in the free-energy so only
the logarithmic term in the Euler function remains. This term exactly cancels the 1/2 from the Ht contribution to
the full free-energy and shows that the learning capacity reaches zero at large sample size.
4.

Normal model unknown mean and variance

The likelihood for the normal model is defined by Eqn. 14 with parameters θ = (~
µ, σ) with support µ
~ ∈ RD and
σ ∈ R>0 . The cross entropy is:
h
i
σ02
2
H(θ; θ0 ) ≡ D
log
2πσ
+
+ 2σ1 2 (~
µ−µ
~ 0 )2 ,
(C27)
2
2
σ
where the true distribution is X ∼ q(~x|θ0 ) and the determinant of the Fisher information matrix is:
det I = 2σ −2(D+1) .

(C28)

The scaled Jeffreys prior (Eqn. 28) is therefore:
ρ=

√

N
(D+1)/2
2( 2πσ
.
2)

(C29)

We will assume w matches the asymptotic form:
w = c σ −D−1 .

(C30)

Note that w must have units of inverse length to the D + 1 power in order to give the evidence the correct units.
Due to translation symmetry in µ, w must be a function of σ only. The partition function is
log Z(X N ) ∼ log c −

DN
2

log 2πσ 2 +

D
2

log

2π
N

− log 2 + log Γ( DN
2 )−

DN
2

log

χ2D(N −1)
2

(C31)

where χ2D(N −1) is a D(N − 1)-dimensional chi-squared random variable. The expected log partition function is:
log Z(θ; N ) = −N H0 (θ) + log c +

DN
2

+

D
2

log

2π
N

− log 2 + log Γ( DN
2 )−

D(N −1)
DN
)
2 ψ(
2

(C32)

24
where H0 is the true entropy and ψ is the polygamma function. The learning capacity is:
C(θ; N, w) =

D
2

2 (1) DN
2 (1) D(N −1)
3 (2) D(N −1)
+ N 2( D
( 2 ) − 2N 2 ( D
( 2 ) − N 3( D
( 2 ),
2) ψ
2) ψ
2) ψ

(C33)

where D is both the dimension of mean parameter and the model. The unknown normalization of w is:
log c =

D+1
2

log

N
2π

+

1
2

log 2 − K

(C34)

written in terms of the effective dimension:
K=

1
2

log

N
2π

−

1
2

log 2 −

DN
2

log

N
N +1

− N log Γ[ D(N2+1) ] + (N + 1) log Γ[ DN
2 ]+

D(N +1)N
2

h
i
D(N −1)
ψ( DN
)) ,
2 − ψ(
2

(C35)

which is used to define the GPI prior w using Eqn. 30.

5.

Exponential model

The likelihood for the normal model is defined:
q(x|θ) ≡ λe−λx

(C36)

which parameters θ ≡ (λ) with support λ ∈ R>0 . The cross entropy is:
H(θ; θ0 ) = − log λ +

λ
λ0 ,

(C37)

where the true distribution is X ∼ q(x|θ0 ) and the determinant of the Fisher information matrix is:
det I = λ−2

(C38)

N
1/2
ρ = ( 2πλ
2)

(C39)

w = c λ−1 .

(C40)

log Z(X N ) ∼ −N log Y + log Γ(N ) + N log λ,

(C41)

The scaled Jeffreys prior (Eqn. 28) is therefore:

We will assume w matches the asymptotic form:

The partition function is

where Y is a Gamma-distributed random variable with unit scale and shape N . The expected log partition function
is:
log Z(θ; N ) = −N H0 (θ) + N (1 − ψ[N ]) + log Γ(N )

(C42)

where H0 is the true entropy and ψ is the polygamma function. The learning capacity is:
C(θ; N ) = N 2 [ψ (1) (N ) − 2ψ (1) (N ) − N ψ (2) (N )],

(C43)

where ψ is the polygamma function. The unknown normalization of w is:
log c =

1
2

log

N
2π

−K

(C44)

which can be re-written as an effective dimension:
K=

1
2

log

N
2π

− N log Γ(N + 1) + (N + 1) log Γ(N ) + N (N + 1)[ψ(N + 1) − ψ(N )],

to define the GPI prior w using Eqn. 30.

(C45)

25
6.

Uniform distribution

The likelihood for the normal model is defined:
(
q(x|θ) ≡

L−1 , 0 ≤ x ≤ L
,
0,
otherwise

where the parameter θ ≡ (L) with support L ∈ R>0 . The cross entropy is:
(
log L, L0 ≤ L
H(θ; θ0 ) =
,
∞,
otherwise

(C46)

(C47)

which is minimized at L = L0 but neither the first nor second derivative is defined at this point and therefore
the Fisher information matrix cannot be defined. We can still infer the dependence of the w by symmetry and
dimensional analysis:
w = c L−1 .

(C48)

log Z(X N ) ∼ log c − N log L − log N − N log Y,

(C49)

The partition function is

where Y is the maximum of N uniformly-distributed random variables on the interval [0, 1]. The CDF for Y is the
N th power of the CDF for a single uniformly-distributed random variable. The expected log partition function is:
log Z(θ; N ) = −N H0 (θ) + log c − log N + 1.

(C50)

C(θ; N ) = 1.

(C51)

log c = log N − N log(1 + N −1 ) − 1,

(C52)

The learning capacity is:

The unknown normalization of w is:

which can be plugged into Eqn. C48 to calculation the GPI prior w.
7.

Poisson stoichiometry problem

Choosing temporal units so that λ = 1, we subdivide our observation time into short times t0  1, for which we
can approximate our Poisson distribution as the joint distribution of Bernoulli trials.
k

q(X N |m) = (mt0 ) (1 − mt0 )

N −k

(C53)

The domain of m is discrete and takes on values {1, 2, . . .} We use this Bernoulli representation of the Poisson process where afterwards we must take the limit where t0 → 0. It is only in this “extensive” representation X N , that
the derivative with respect to sample size is a predictive density. However, we can to perform thermodynamic
calP
culations without this limiting procedure using the sufficient statistic likelihood p(k|θ), where k = i xi following
the reasoning laid out in B 3. The sufficient statistic partition function as a function of k is
z(k; t) =

∞
X
m=1

e−mt

(mt)k
$(m).
k!

(C54)

The likelihood with the one parameter prior $(m) = e−bm . This can be recast using Poisson resummation into an
equivalent sum
tk
z(k; t) =
(b + t)k+1

−
∞ 
X
4π 2 ν 2
1+2
+
1
(b + t)2
ν=1

k+1
2



!
2πν
cos (k + 1)arctan
b+t

(C55)

26

log-prior weight: log( )

0
GPI prior
Jeffreys prior

-1
-2
-3
-4

0.0

1.0

2.0

3.0

log-fluorophore number: log(m)
FIG. 9. Jeffreys prior and GPI prior at t = 1 The power law behavior of the GPI prior and the Jeffreys prior (m−1/2 ) for the
Poisson problem at t = 1 are compared on a log-log plot. At large fluorophore number, the discrete problem is very similar to
the continuous problem, and the GPI prior converges to the same power law behavior as the Jeffreys prior. At small sample
size, effects from the discretization deform the GPI prior away from the Jeffreys prior. The normalization of the Jeffreys prior
has been chosen to make the two priors match at large m.
Model

Parameter support

Generative parameters

θ

θ0
µ0 = 5, σ0 = 1

N
µ ∈ [0, 10]

µ0 = 6, σ0 = 1

N(µ, σ)

µ ∈ [0, 10], σ ∈ [0.1, 10]

µ0 = 5, σ0 = 0.75

Exp(λ)

λ ∈ [0.1, 10]

λ0 = 2

L ∈ [0, 10]

L0 = 10

N(µ)

U(L)

TABLE IV. Models for inference on simulated data with revised support. Five data sets were generated, one for each model,
using the generative parameters: X N ∼ q(·|θ0 ). Inference was performed on the simulated data using the GPI prior w.

The first expression is useful for large k, when the width of the likelihood is much smaller than the unit spacing.
The second one can be viewed as an expansion from the approximation when the sum is replaced by an integral,
it is therefore useful when the posterior is of large or moderate width (k/t2 > .1).
Finally, we also have the closed form recursion relation which is useful for small k
z(k; t) =

−t∂t
z(k − 1; t)
k

with

z(0; t) = (eb+t − 1)−1

(C56)

when $(m) = e−bm . This is useful when k is small. When k is large, this closed expression grows into sums
with combinatorial coefficients. It is then from the disorder averaged free energy that the partition functions can
be calculated after the sample size derivatives are corrected for the changing point entropy which we describe in
App. B 3
F (m0 ) − H0 = −t E log zt (k) + st (m0 )
k|m0

(C57)

where st is the entropy of the count distributions.
Using the expressions for the entropy, and the recursive scheme described in App. B 4, we can construct the
w-prior. The Jeffreys prior obeys power law scaling for the continuous case [55–57]. In the small-sample size limit
we recover the standard Jeffreys prior at moderate sized fluorophore number as shown inf Fig. C 7, while in the
large sample-size limit we converge to the Laplace principle of indifference: each discrete value of the parameter
assigned equal, in fact unit, weight.

[1] Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.

27
[2] E. T. Jaynes. Probability Theory: The Logic of Science. Cambridge University Press., 2003.
[3] Ole E Barndorff-Nielsen and Peter E Jupp. Statistics, yokes and symplectic geometry. In Annales de la Faculté des sciences de
Toulouse: Mathématiques, volume 6, pages 389–427, 1997.
[4] V. Balasubramanian. Statistical inference, Occam’s razor, and statistical mechanics on the space of probability distributions.
Neural Computation, 9:349–368, 1997.
[5] Hidetoshi Nishimori. Statistical physics of spin glasses and information processing: an introduction, volume 111. Clarendon
Press, 2001.
[6] Marc Mezard and Andrea Montanari. Information, physics, and computation. Oxford University Press, 2009.
[7] Arnaud Doucet, Simon Godsill, and Christophe Andrieu. On sequential monte carlo sampling methods for bayesian
filtering. Statistics and computing, 10(3):197–208, 2000.
[8] Matthew D Hoffman and Andrew Gelman. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte
carlo. Journal of Machine Learning Research, 15(1):1593–1623, 2014.
[9] Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International Conference on
Machine Learning, pages 1683–1691, 2014.
[10] Thomas P Minka. Expectation propagation for approximate bayesian inference. In Proceedings of the Seventeenth conference
on Uncertainty in artificial intelligence, pages 362–369. Morgan Kaufmann Publishers Inc., 2001.
[11] Eric P Xing, Michael I Jordan, and Stuart Russell. A generalized mean field algorithm for variational inference in exponential families. In Proceedings of the Nineteenth conference on Uncertainty in Artificial Intelligence, pages 583–591. Morgan
Kaufmann Publishers Inc., 2002.
[12] Tom Minka et al. Divergence measures and message passing. Technical report, Technical report, Microsoft Research, 2005.
[13] Teppo Mikael Niinimäki and Mikko Koivisto. Annealed importance sampling for structure learning in bayesian networks.
In IJCAI, pages 1579–1585, 2013.
[14] Faming Liang and Wing Hung Wong. Real-parameter evolutionary monte carlo with applications to bayesian mixture
models. Journal of the American Statistical Association, 96(454):653–666, 2001.
[15] Jun-ichi Inoue. Application of the quantum spin glass theory to image restoration. Physical Review E, 63(4):046114, 2001.
[16] Marc Mézard and Andrea Montanari. Reconstruction on trees and spin glass transition. Journal of statistical physics,
124(6):1317–1350, 2006.
[17] Lenka Zdeborová and Florent Krzakala. Statistical physics of inference: Thresholds and algorithms. Advances in Physics,
65(5):453–552, 2016.
[18] R.K. Pathria and P.D. Beale. Statistical Mechanics. Elsevier Science, 1996.
[19] Benjamin B Machta, Ricky Chachra, Mark K Transtrum, and James P Sethna. Parameter space compression underlies
emergent theories and predictive models. Science, 342(6158):604–7, Nov 2013.
[20] Josiah Willard Gibbs. Elementary principles of statistical mechanics. Compare, 289:314, 1902.
[21] S Bernstein. Theory of probability, 1927.
[22] Lucien LeCam. On some asymptotic properties of maximum likelihood estimates and related bayes estimates. Univ.
California Pub. Statist., 1:277–330, 1953.
[23] Jayanta K Ghosh, Subhashis Ghosal, and Tapas Samanta. Stability and convergence of the posterior in non-regular problems. Statistical Decision Theory and Related Topics V, page 183, 2012.
[24] S. Watanabe. Algerbraic geometry and statistical learning theory. Cambridge Univeristy Press, 2009.
[25] J. Rissanen. Modeling by the shortest data description. Automatica, 14:465–471, 1978.
[26] Peter D. Grünwald. The Minimum Description Length Principle. MIT, Cambridge, MA, 2007.
[27] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer Series in Statistics.
Springer New York Inc., New York, NY, USA, 2001.
[28] H. Akaike. Information theory and an extension of the maximum likelihood principle. In Petrov B. N. and E. Csaki, editors,
2nd International Symposium of Information Theory., pages 267–281. Akademiai Kiado, Budapest., 1973.
[29] K. P. Burnham and D. R. Anderson. Model selection and multimodel inference. Springer-Verlag New York, Inc., 2nd. edition,
1998.
[30] P. S. Laplace. Theorie analytique des probabilites. Courcier Imprimeur, Paris, 3rd edition.
[31] J. M. Keynes. A Treatise on Probability. Macmillan Limited, London, 1921.
[32] J. Shore and R. Johnson. Axiomatic derivation of the principle of maximum entropy and the principle of minimum crossentropy. Information Theory, IEEE Transactions on, 26(1):26–37, January 1980.
[33] R. L. Kashyap. Prior probability and uncertainty. IEEE Transactions on information theory, 17(6):641–650, 1971.
[34] F. Reif. Statistical Physics. McGraw-Hill (New York), 1967.
[35] Duane C Wallace. Statistical physics of crystals and liquids: a guide to highly accurate equations of state. World Scientific, 2002.
[36] Brent Fultz. Vibrational thermodynamics of materials. Progress in Materials Science, 55(4):247–352, 2010.
[37] Leo A Goodman. Some practical techniques in serial number analysis. Journal of the American Statistical Association,
49(265):97–112, 1954.
[38] Robert E Kass and Larry Wasserman. The selection of prior distributions by formal rules. Journal of the American Statistical
Association, 1996.
[39] H. Jeffreys. An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London.
Series A, Mathematical and Physical Sciences, 186(1007):453–461, 1946.
[40] J. M. Bernardo. Bayesian statistics 6: Nested Hypothesis Testing: The Bayesian Reference Criterion. Oxford University Press,
1999.
[41] James O. Berger, Jose M. Bernardo, and Dongchu Sun. Objective priors for discrete parameter spaces. Journal Of The
American Statistical Association, 107(498):636–648, 2012.

28
[42] Andrew Gelman, Jessica Hwang, and Aki Vehtari. Understanding predictive information criteria for bayesian models.
Statistics and Computing, 24(6):997–1016, 2014.
[43] Alan E Gelfand and Dipak K Dey. Bayesian model choice: asymptotics and exact calculations. Journal of the Royal Statistical
Society. Series B (Methodological), pages 501–514, 1994.
[44] D. J. Spiegelhalter, N. G. Best, B. P. Carlin, and A. van der Linde. Bayesian measures of model complexity and fit (with
discussion). Journal of the Royal Statistical Society, B64:583–639, 2002.
[45] Aki Vehtari and Janne Ojanen. A survey of Bayesian predictive methods for model assessment, selection and comparison.
Statistics Surveys, 6:142–228, 2012.
[46] Kenneth P. Burnham and David R. Anderson. Multimodel inference: Understanding AIC and BIC in model selection.
Socialogical Methods & Research, 33(2):261–304, 2004.
[47] Mark K Transtrum, Benjamin B Machta, Kevin S Brown, Bryan C Daniels, Christopher R Myers, and James P Sethna.
Perspective: Sloppiness and emergent theories in physics, biology, and beyond. J Chem Phys, 143(1):010901, Jul 2015.
[48] M. Stone. An asymptotic equivalence of choice of model by cross-validation and Akaike’s Criterion. Journal of the Royal
Statistical Society. Series B (Methodological), 39(1):44–47., 1977.
[49] D. V. Lindley. A statistical paradox. Biometrika, 44(1/2):187–192, 1957.
[50] M. S. Bartlett. A comment on D. V. Lindley’s statistical paradox. Biometrika, 44(3/4):533–534, 1957.
[51] J. O. Berger and J.-M. Bernardo. On the development of the reference prior method. Techincal Report 91-15C, Purdue
University, 1991.
[52] Robert D. Cousins. The Jeffreys–Lindley paradox and discovery criteria in high energy physics. Synthese, pages 1–38, 2014.
[53] Colin H. LaMont and Paul A. Wiggins. The lindley paradox: The loss of resolution in bayesian inference. Unbder review.
(arXiv:1610.09433), 2017.
[54] Jun Shao. An asymptotic theory for linear model selection. Statistica Sinica, 7:221–264, 1997.
[55] Edwin T Jaynes. Prior probabilities. IEEE Transactions on systems science and cybernetics, 4(3):227–241, 1968.
[56] Keegan E Hines. Inferring subunit stoichiometry from single molecule photobleaching. The Journal of general physiology,
141(6):737–746, 2013.
[57] Timothy D Ross. Accurate confidence intervals for binomial proportion and poisson rate estimation. Computers in biology
and medicine, 33(6):509–531, 2003.

