Maximizing the information learned from finite
data selects a simple model
Henry H. Mattinglya , Mark K. Transtrumb , Michael C. Abbottc,1 , and Benjamin B. Machtad,1
a
Lewis-Sigler Institute and Department of Chemical and Biological Engineering, Princeton University, Princeton, NJ 08544, USA; b Department of Physics and Astronomy,
Brigham Young University, Provo, Utah 84602, USA; c Marian Smoluchowski Institute of Physics, Jagiellonian University, Ulica Łojasiewicza 11, 30-348 Kraków, Poland;
d
Lewis-Sigler Institute and Department of Physics, Princeton University, Princeton, NJ 08544, USA

arXiv:1705.01166v3 [physics.data-an] 14 Feb 2018

PNAS (7 February 2018) + SI ≈ arXiv:1705.01166v3
Earlier versions had the title “Rational Ignorance: Simpler Models Learn More Information from Finite Data”

We use the language of uninformative Bayesian prior choice to study
the selection of appropriately simple effective models. We advocate
for the prior which maximizes the mutual information between parameters and predictions, learning as much as possible from limited data.
When many parameters are poorly constrained by the available data,
we find that this prior puts weight only on boundaries of the parameter manifold. Thus it selects a lower-dimensional effective theory in a
principled way, ignoring irrelevant parameter directions. In the limit
where there is sufficient data to tightly constrain any number of parameters, this reduces to Jeffreys prior. But we argue that this limit is
pathological when applied to the hyper-ribbon parameter manifolds
generic in science, because it leads to dramatic dependence on effects invisible to experiment.

The theory and the experiment are together described by a
probability distribution p(x|θ), for each value of the theory’s
parameters θ ∈ Θ. This function encodes both the quality and
quantity of data to be collected.
The mutual information between the parameters and their
expected data is defined as MI = I(X; Θ) = S(Θ) − S(Θ|X),
where S is the Shannon entropy (23). The MI thus quantifies
the information which can be learned about the parameters
by measuring the data, or equivalently, the information about
the data which can be encoded in the parameters (24, 25).
Defining p? (θ) by maximizing this, we see:
i. The prior p? (θ) is almost always discrete (26–30), with
weight only on a finite number
PK K of points, or atoms
(Figures 1 and 2): p? (θ) = a=1 λa δ(θ − θa ).

Effective Theory | Model Selection | Renormalization Group | Bayesian
Prior Choice | Information Theory

ii. When data is abundant, p? (θ) approaches Jeffreys prior
pJ (θ) (31–33). As this continuum limit is approached,
the proper spacing of the atoms shrinks as a power law
(Figure 3).

P

hysicists prefer simple models not because nature is simple, but because most of its complication is usually irrelevant. Our most rigorous understanding of this idea comes from
the Wilsonian renormalization group (1–3), which describes
mathematically the process of zooming out and losing sight
of microscopic details. These details only influence the effective theory which describes macroscopic observables through
a few relevant parameter combinations, such as the critical
temperature, or the proton mass. The remaining irrelevant
parameters can be ignored, as they are neither constrained
by past data nor useful for predictions. Such models can now
be understood as part of a large class called sloppy models
(4–14), whose usefulness relies on a similar compression of a
large microscopic parameter space down to just a few relevant
directions.
This justification for model simplicity is different from the
one more often discussed in statistics, motivated by the desire
to avoid overfitting (15–21). Since irrelevant parameters have
an almost invisible effect on predicted data, they cannot be
excluded on these grounds. Here we motivate their exclusion
differently: we show that simplifying a model can often allow it
to extract more information from a limited data set, and that
this offers a guide for choosing appropriate effective theories.
We phrase the question of model selection as part of the
choice of a Bayesian prior on some high-dimensional parameter space. In a set of nested models, we can always move
to a simpler model by using a prior which is nonzero only on
some subspace. Recent work has suggested that interpretable
effective models are typically obtained by taking some parameters to their limiting values, often 0 or ∞, thus restricting to
lower-dimensional boundaries of the parameter manifold (22).
Our setup is that we wish to learn about a theory by
performing some experiment which produces data x ∈ X.

iii. When data is scarce, most atoms lie on boundaries of
parameter space, corresponding to effective models with
fewer parameters (Figure 4). The resulting distribution of
weight along relevant directions is much more even than
that given by Jeffreys prior (Figure 5).

Significance Statement
Most physical theories are effective theories, descriptions at
the scale visible to our experiments which ignore microscopic
details. Seeking general ways to motivate such theories, we
find an information theory perspective: if we select the model
which can learn as much information as possible from the
data, then we are naturally led to a simpler model, by a path
independent of concerns about overfitting. This is encoded as
a Bayesian prior which is nonzero only on a subspace of the
original parameter space. We differ from earlier prior selection
work by not considering infinitely much data. Having finite data
is always a limit on the resolution of an experiment, and in our
framework this selects how complicated a theory is appropriate.
H.H.M. and M.C.A. performed the numerical experiments. H.H.M., M.K.T., M.C.A. and B.B.M. designed the research, interpreted results, and contributed to writing. M.C.A. and B.B.M. led the
writing of the paper.
Present affiliation for H.H.M.: Department of Molecular Cellular and Developmental Biology, Yale
University, New Haven, CT, 06520
Present affiliations for B.B.M: Department of Physics, Yale University, New Haven, CT, 06520
Systems Biology Institute, Yale University, West Haven, CT, 06516
1

To whom correspondence should be addressed. E-mail: abbott@th.if.uj.edu.pl and
benjamin.machta@yale.edu

A

B

C

Fig. 1. Optimal priors for the Bernoulli model (1). Red lines indicate the positions
of delta functions in p? (θ), which are at the maxima of fKL (θ), Eq. (3). As
m → ∞ these coalesce into Jeffreys prior pJ (θ).

After some preliminaries, we demonstrate these properties in
three simple examples, each a stylized version of a realistic
experiment. To see the origin of discreteness, we study the bias
of an unfair coin and the value of a single variable corrupted
with Gaussian noise. To see how models of lower dimension
arise, we then study the problem of inferring decay rates in a
sum of exponentials.
In the Appendix we discuss the alorithms used for finding
p? (θ) (Figures 2 and S3), and we apply some more traditional
model selection tools to the sum of exponentials example
(Figure S1).

Priors and Geometry
Bayes’ theorem tells us how to update our knowledge of θ
upon observing data x, from prior
p(θ) to posterior p(θ|x) =
R
p(x|θ)p(θ)/p(x), where p(x) = dθ p(θ) p(x|θ). In the absence
of better knowledge we must pick an uninformative prior
which codifies our ignorance. The naive choice of a flat prior
p(θ) = const. has undesirable features, in particular making
p(x) depend on the choice of parameterization, through the
measure dθ.
Jeffreys prior pJ (θ) is invariant under changes of parameterizaion, because it is constructedpfrom some properties of
the experiment (34). This pJ (θ) ∝ det gµν is, up to normalization, the volume form arising from the Fisher information
metric (FIM, often -matrix):

Bernardo defined a prior p? (θ) by maximizing the mutual
information between parameters Θ and the expected data X m
from m repetitions, and then a reference prior by taking the
limit m → ∞ (29, 31). Under certain benign assumptions,
this reference prior is exactly Jeffreys prior (31–33), providing
an alternative justification for pJ (θ).
We differ in taking seriously that the amount of data collected is always finite.∗ Besides being physically unrealistic,
the limit m → ∞ is pathological both for model selection
and prior choice. In this limit any number of parameters
can be perfectly inferred, justifying an arbitrarily complicated
model. In addition, in this limit the posterior p(θ|x) becomes
independent of any smooth prior.†
Geometrically, the defining feature of sloppy models is that
they have a parameter manifold with hyper-ribbon structure
(6–9): there are some long directions (corresponding to d relevant, or stiff, parameters) and many shorter directions (D − d
irrelevant, or sloppy, parameter combinations). These lengths
are often estimated using the eigenvalues of gµν , and have
logarithms that are roughly evenly-spaced over many orders
of magnitude (4, 5). The effect of coarse-graining is to shrink
irrelevant directions (here using the technical meaning of irrelevant: a parameter which shrinks under renormalization group
flow) while leaving relevant directions extended, producing a
sloppy manifold (8, 14). By contrast the limit m → ∞ has the
effect of expanding all directions, thus erasing the distinction
between directions longer and shorter than the critical length
scale of (approximately) one standard deviation.
On such a hyper-ribbon, Jeffreys prior has an undesirable
feature: since it is constructed from the D-dimensional notion
of volume, its weight along the relevant directions always
depends on the volume of the D − d irrelevant directions. This
gives it extreme dependence on which irrelevant parameters
are included in the model.‡ The optimal prior p? (θ) avoids
this dependence because it is almost always discrete, at finite
m.§ It puts weight on a set of nearly distinguishable points,
closely spaced along the relevant directions, but ignoring the
irrelevant ones. Yet being the solution to a reparameterizationinvariant optimization problem, the prior p? (θ) retains this
good feature of pJ (θ).
Maximizing the mutual information was originally done to
calculate the capacity of a communication channel, and we
can borrow techniques from rate-distortion theory here: the
algorithms we use were developed there (37, 38), and the discreteness we exploit was discovered several times in engineering
(26–28, 39). In statistics, this problem is more often discussed
as an equivalent minimax problem (40). Discreteness was
∗

~ =
gµν (θ)

Z

~
~
~ ∂ log p(x|θ) ∂ log p(x|θ) .
dx p(x|θ)
µ
ν
∂θ
∂θ

This Riemannian metric defines a P
reparameterization-invariant
D
distance between points, ds2 =
g dθµ dθν . It meaµ,ν=1 µν
sures the distinguishability of the data which θ and θ + dθ are
expected to produce, in units of standard deviations. Repeating an (identical and independently distributed)
experiment
Qm
m times means considering pm (~
x|θ) = j=1 p(xj |θ), which
m
leads to metric gµν
(θ) = m gµν (θ). However the factor mD/2
in the volume is lost by normalizing pJ (θ). Thus Jeffreys prior
depends on the type of experiment, but not the quantity of
data.

PNAS

|

7 February, 2018

|

vol. XXX

|

no. XX

|

2

Interned for five years, John Kerrich only flipped his coin 104 times (35). With
computers we can do better, but even the LHC only generated about 1018 bits of
data (36).
†
For simplicity we consider only regular models, i.e. we assume all parameters are
structurally identifiable.
‡
See Figure 5 for a demonstration of this point. For another example, consider a
parameter manifold Θ which is a cone, with Fisher metric ds2 = (50 dϑ)2 +
ϑ2 dΩ2n /4: there is one relevant direction ϑ ∈ [0, 1] of length L = 50, and n
irrelevant directions forming a sphere of diameter ϑ. Then the prior on ϑ alone
~ is p(ϑ) = (n + 1)ϑn , putting most of the weight near to ϑ = 1,
implied by pJ (θ)
dramatically so if n = D − d is large. But since only the relevant direction is visible
to our experiment, the region ϑ ≈ 0 ought to be treated similarly to ϑ ≈ 1. The prior
~ has this property.
p? (θ)
§
We offer both numerical and analytic arguments for discreteness below. The exception
to discreteness is that if there is an exact continuous symmetry, p? (θ) will be constant
along it. For example if our Gaussian model Eq. (2) is placed on a circle (identifying
both θ ∼ θ + 1 and x ∼ x + 1) then the optimum prior is a constant.

Mattingly, Transtrum, Abbott, Machta

|

Rational Ignorance

also observed in other minimax problems (41–43), and later
in directly maximising mutual information (29, 30, 33, 44).
However it does not seem to have been seen as useful, and
none of these papers explicitly find discrete priors in dimension
D > 1, which is where we see attractive properties. Discreteness has been useful, although for different reasons, in the idea
of rational inattention in economics (45, 46). There, market
actors have a finite bandwidth for news, and this drives them
to make discrete choices despite all the dynamics being continuous. Rate-distortion theory has also been useful in several
areas of biology (47–49), and discreteness emerges in a recent
theoretical model of the immune system (50).
We view this procedure of constructing the optimal prior
as a form of model selection, picking out the subspace of Θ
on which p? (θ) has support. This depends on the likelihood
function p(x|θ) and the data space X, but not on the observed
data x. In this regard it is closer to Jeffreys’ perspective
on prior selection than to tools like the information criteria
and Bayes factors, which are employed at the stage of fitting
to data. We discuss this difference at more length in the
Appendix.

log p𝜏(𝜃)

𝜎 = 1/10

2
2
1
e−(x−θ) /2σ .
p(x|θ) = √
2πσ

Mattingly, Transtrum, Abbott, Machta

|

Rational Ignorance

𝜃, with steps Δ𝜃 = 1/300

�

�

log p⭑(𝜃)



dθ p(θ) fKL (θ),



[3]

Z

fKL (θ) = DKL p(x|θ) p(x) =

dx p(x|θ) log

p(x|θ)
p(x)

¶
where DKL is the Kullback–Leibler divergence.
MaximizR
ing MI over all functions p(θ) with dθ p(θ) = 1 gives
fKL (θ) = const. But the maximizing function will not, in
general, obey p(θ) ≥ 0. Subject to this inequality p? (θ) must
satisfy

[1]

{p? (θ) > 0, fKL (θ) = MI} or {p? (θ) = 0, fKL (θ) < MI}
at every θ. With finite data fKL (θ) − MI must be an analytic
function of θ, and therefore must be smooth with a finite
numbers of zeros, corresponding to the atoms of p? (θ) (Figure
1A). See (28, 29, 46) for related arguments for discreteness,
and (41–43) for other approaches.
The number of atoms occurring in p? (θ) increases as the
data improves. For K atoms there is an absolute bound
MI ≤ log K, saturated if they are perfectly distinguishable. In
Figure 2C we observe that the optimal priors instead approach
a line MI → ζ log K, with slope ζ ≈ 0.75. At large L the
length of parameter space is proportional to the number of
distinguishable points, hence MI → log L. Together these
imply K ∼ L1/ζ , and so the average number density of atoms
grows as
ρ0 = K/L ∼ L1/ζ−1 ≈ L1/3 ,

[2]

Repeated
measurements are equivalent to smaller σ (by σ →
√
σ/ m), so we fix m = 1 here. The Fisher metric is gθθ = 1/σ 2 ,
thus L = 1/σ. An optimal prior is shown in Figure 2, and
in Figure 5A along with its implied distribution of expected
data. This p(x) is similar to that implied by Jeffreys prior,
here pJ (θ) = 1.
We calculated p? (θ) numerically in two ways. After discretizing both θ and x, we can use the Blahut–Arimoto (BA)

�

Z
MI = I(X; Θ) =

p

m
,
θ(1−θ)

𝜃, with steps Δ𝜃 = 1/30

◼

◼

algorithm (37, 38). This converges to the global maximum,
which is a discrete distribution: see Figure 2. Alternatively,
using our knowledge that p? (θ) is discrete, we can instead
adjust the positions θa and weights λa of a finite number of
atoms. See the Supplement for more details.
To see analytically why discreteness arises, we write the
mutual information as

thus pJ (θ) = [π θ(1 − θ)]−1 , and
R√
√
proper parameter space length L =
ds2 = π m.
In the extreme case m = 1, the optimal prior is two delta
functions, p? (θ) = 12 δ(θ) + 12 δ(θ − 1), and MI = log 2, exactly
one bit (29, 30, 33). Before an experiment that will only
ever be run once, this places equal weight on both outcomes;
afterwards it records the outcome. As m increases, weight is
moved from the boundary onto interior points, which increase
in number, and ultimately approach the smooth pJ (θ): see
Figures 1 and 2A.
Similar behavior is seen in a second example, in which we
measure one real number x, normally distributed with known
σ about the parameter θ ∈ [0, 1]:

This gives gθθ =

◼

Fig. 2. Convergence of the Blahut–Arimoto algorithm. This is for the oneparameter Gaussian model Eq. (2) with L = 10 (comparable to m = 10 in
Figure 1). On the right θ is discretized into ten times as many points, but pτ (θ)
clearly converges to the same five delta functions.

We begin with some problems with a single bounded parameter, of length L in the Fisher metric. These tractable cases
illustrate the generic behaviour along either short (irrelevant)
or long (relevant, L  1) parameter directions in higherdimensional examples.
Our first example is the Bernoulli problem, in which we
wish to determine the probability θ ∈ [0, 1] that an unfair
coin gives heads, using the data from m trials. It is sufficient
to record the total number of heads x, which occurs with
probability
m!
θx (1 − θ)m−x .
x!(m − x)!

◼

◼

�

One-Parameter Examples

p(x|θ) =

𝜏 = 10 iterations
𝜏 = 100
𝜏 = 104
Exact p⭑(𝜃)

¶

L  1.

[4]

The function fKL (θ) is sometimes called the Bayes risk, as it quantifies how poorly
the prior will perform if θ turns out to be correct. One of the problems equivalent
to maximising the mutual information (40) is the minimax problem for this (see also
Figure 1):

max I(X; Θ) = min max fKL (θ) = min max
p(θ)

p(θ)

θ

R

dθp(θ)DKL [p(x|θ)kq(x)] .

q(x) p(θ)

The distributions we call expected data p(x) are also known as Bayes strategies, i.e.
distributions on X which are the convolution of the likelihood p(x|θ) with some prior
p(θ). The optimal q(x) from this third formulation (with minq(x) . . .) can be shown
to be such a distribution (40).

PNAS

|

7 February, 2018

|

vol. XXX

|

no. XX

|

3

B

𝜃

r=

A

0

B

A

p
Bernoulli: L = ⇡ m

r=
=

k2

Gaussian: L = 1/𝜎

k1

C

1

y2

s = 1ê3

s = 1ê7

y1
‡

0.5

Ê‡‡
Ê‡‡
Ê‡ÊÊÊ
‡‡
‡‡‡‡‡‡‡‡‡ ‡ ‡ ‡‡‡‡‡‡‡‡‡‡‡‡‡‡‡
‡‡ÊÊÊ
‡‡
‡‡
‡‡
‡‡
‡‡‡‡
‡
‡‡
‡
‡‡
‡‡
‡‡
‡‡
‡‡
‡‡‡
Ê
Ê
‡‡
‡‡
Ê‡Ê
‡Ê‡
ÊÊ
Ê

Ê

d eff

𝜁=1
𝜁 = 3/4

1
0

‡

‡
‡‡‡‡ ‡ ‡
‡‡‡‡
‡‡‡‡

2

Number of atoms K

Ê
ÊÊÊ
‡Ê‡
Ê‡Ê
Ê‡‡
Ê‡‡
‡‡‡‡
‡‡‡‡
‡‡‡
‡‡‡

5

Ê

Ê

rá2
Ê
Ê

rá1

rá0

0
2

Ê

Ê

Ê

Bernoulli
Gaussian

Ê

10 20
1ês

Ê

Ê

Ê

Ê

ÊÊ
Ê

s = 1ê19

2

Wr

I(X; ⇥)/ log 2

1

Ê

Ê

Ê Ê
Ê ÊÊ

Ê

ÊÊ
Ê Ê

r=

C

Ê Ê ÊÊ
Ê ÊÊÊÊ

Ê
ÊÊ
Ê Ê

Ê
Ê ÊÊ

s = 1ê50
50 100

1/𝜎

Fig. 3. Behavior of p? (θ) with increasing Fisher length. Panels A and B show
the atoms of p? (θ) for the two one-dimensional models as L is increased (i.e. we
perform more repetitions m or have smaller noise σ ). Panel C shows the scaling of
the mutual A
information (in bits) with the number of atoms K .The dashed line is the
bound MI ≤ log K , and the solid line is the scaling law MI ∼ 3/4 log K .

Thus the proper spacing between atoms shrinks to zero in
the limit of infinite data, i.e. neighboring atoms cease to be
distinguishable.
To derive this scaling law analytically, in a related paper
(51) we consider a field theory for the number density of atoms,
in which the entropy density (omitting numerical factors) is

B

2

S = const. − e−ρ [ρ4 (ρ0 )2 + 1].

From this we find ζ = 3/4, which is consistent with both
examples presented above.

Multi-parameter Example
In the examples above, p? (θ) concentrates weight on the edges
of its allowed domain when data is scarce (i.e. when m is
small or σ is large, hence L is small). We next turn to a multiparameter model in which some parameter combinations are
ill-constrained, and where edges correspond to reduced models.
The physical picture is that we wish to determine the
composition of an unknown radioactive source, from data of xt
Geiger counter clicks at some times t. As parameters we have
the quantities Aµ and decay constants kµ of isotopes µ. The
probability of observing xt should be a Poisson distribution (of
mean yt ) at each time, but we approximate these by Gaussians
to write:‖
p(~x|~
y) ∝

Y

2

e−(xt −yt )

/2σ 2

,

X

yt =

t
‖

Aµ e−kµ t .

[5]

µ

Using a normal distribution of fixed σ here is what allows the metric in Eq. (6) to
be so simple. However the qualitative behavior from the Poisson distribution is very
similar.

PNAS

|

7 February, 2018

|

vol. XXX

|

no. XX

|

4

Fig. 4. Parameters and priors for the exponential model (5). Panel A shows the
area of the ~
y plane covered by all decay constants k1 , k2 ≥ 0. Panel B shows the
positions of the delta functions of the optimal prior p? (~
y ) for several values of σ , with
colors indicating the dimensionality r at each point. Panel C shows the proportion of
weight on these dimensionalities.

Fixing σt = σ = const. then brings us to a nonlinear leastsquares model of the type studied by (6, 7). This same model
also arises in other contexts, such as the asymptotic approach
to equilibrium of many dynamical systems (52).
We can see the essential behavior with just two isotopes
in fixed quantities: Aµ = 12 , thus yt = 12 (e−k1 t + e−k2 t ).
Measuring at only two times t1 and t2 , we almost have a
two-dimensional version of Eq. (2), in which the center of the
distribution ~
y = (y1 , y2 ) plays the role of θ above. The mapping between ~k and ~
y is shown in Figure 4A, fixing t2 /t1 = e.
The Fisher information metric is proportional to the ordinary
Euclidean metric for ~
y , but not for ~k:
gµν (~k) =

1 X ∂yt ∂yt
σ2
∂kµ ∂kν

⇐⇒

gst (~
y) =

t

1
δst .
σ2

[6]

Thus Jeffreys prior is a constant on the allowed region of the
~
y plane.
Then we proceed to find the optimum p? (~
y ) for this model,
shown in Figure 4B for various values of σ. When σ is large,
this has delta functions only in two of the corners, allowing
only k1 , k2 = 0 and k1 , k2 = ∞. As σ is decreased, new atoms
appear first along the lower boundary (corresponding to the
one-dimensional model where k1 = k2 ) and then along the
other boundaries. At sufficiently small σ, atoms start filling
in the (two-dimensional) interior.
To show this progression in Figure 4C, we define Ωr as
the total weight on all edges
PD of dimension r, and an effective
dimensionality deff =
r Ωr . This increases smoothly
r=1
from 0 towards D = 2 as the data improves.
Mattingly, Transtrum, Abbott, Machta

|

Rational Ignorance

A

Gaussian model: D = 1
σ = �/��

p(x) from p⭑(𝜃)

�★ (θ)

p(x) from Jeffreys

■

■
■

■

■

Discussion

𝜃
θ

�

B

Exponential
model: D = 2

�

Prior p? (~y)

�★ (�+ )

y+ =

y1 + y2+
p
2

s = 1ê7

σ = �/�

r=0
r=1
■

■
■

■
■

�

to equal decay constants will still have significant weight.
The fact that such edges give human-readable simpler models
(unlike arbitrary submanifolds) was the original motivation
for preferring them in (22), and it is very interesting that our
optimization procedure has the same preference.††

■

0

■

y1 + y2
y+ = p
�+ = (�� +�2� )/ �

�

Fig. 5. Distributions of expected data p(x) from different priors. Panel A is the
one-parameter Gaussian model, with L = 10. Panel B projects the two-parameter
exponential model onto the y1 + y2 direction, for σ = 1/7 where the perpendicular
direction should be irrelevant. The length
is about the same
√ of the relevant direction
p? (~y) of expected
as the one-parameter case: L+ = 7 2. Notice that the distribution
data p(x+ ) from Jeffreys
√prior here is quite different, with almost no weight at the
ends of the range (0 and 2), because this prior still weights the area not the length.

At medium values of σ, the prior p? (~
y ) almost ignores the
width of the parameter
manifold,
and
cares
mostly about its
√
length (L+ = 2/σ along the diagonal). This behavior is very
different to that of Jeffreys prior: in Figure 5B we demonstrate
this by plotting the distributions of data implied by these two
priors. Jeffreys puts almost no weight near the ends of the long
(i.e. stiff, or relevant) parameter’s range, because the (sloppy,
or irrelevant) width happens to be even narrower there than
in the middle. By contrast our effective model puts significant
weight on each end, much like the one-parameter model in
Figure 5A.
The difference between one and two parameters being relevant (in Figure 4B) is very roughly σ = 1/7 to σ = 1/50, a
factor 7 in Fisher length, thus a factor 50 in the number of repetitions m — perhaps the difference between a week’s data and
a year’s. These numbers are artificially small to demonstrate
the appearance of models away from the boundary: more
realistic models often have manifold lengths spread over many
orders of magnitude (5, 8), and thus have some parameters
inaccessible even with centuries of data. To measure these we
need a qualitatively different experiment, justifying a different
effective theory.
The one-dimensional model along the lower edge of Figure
4A is the effective theory with equal decay constants. This
remains true if we allow more parameters k3 , k4 , . . . in Eq. (5),
and p? (~
y ) will still place a similar weight there.∗∗ Measuring xt
also at later times t3 , t4 , . . . will add more thin directions to the
manifold (7), but the one-dimensional boundary corresponding

While the three examples we have studied here are very simple,
they demonstrate a principled way of selecting optimal effective
theories, especially in high-dimensional settings. Following
(45), we may call this rational ignorance.
The prior p? (θ) which encodes this selection is the maximally uninformative prior, in the sense of leaving maximum
headroom for learning from data. But its construction depends on the likelihood function p(x|θ), and thus it contains
knowledge about the experiment through which we are probing
nature. Jeffreys prior pJ (θ) also depends on the experiment,
but more weakly: it is independent of the number of repetitions m, precisely because it is the limit m → ∞ of the
optimal prior (32, 33).
Under either of these prescriptions, performing a second
experiment may necessitate a change in the prior, leading to
a change in the posterior not described by Bayes’ theorem. If
the second experiment is different from the first, then changing to Jeffreys prior for the combined experiment (and then
applying Bayes’ rule just once) will have this effect (56, 57).‡‡
Our prescription differs from Jeffreys in also regarding more
repetitions of an identical experiment as being different. Many
experiments would have much higher resolution if they could
be repeated for all eternity. The fact that they cannot is an
important limit on the accuracy of our knowledge, and our
proposal treats this limitation on the same footing as the rest
of the specification of the experiment.
Keeping m finite is where we differ from earlier work on
prior selection. Bernardo’s reference prior (31) maximizes the
same mutual information, but always in the m → ∞ limit
where it gives a smooth analytically tractable function. Using
I(X; Θ) to quantify what can be learned from an experiment
goes back to Lindley (24). That finite information implies a
discrete distribution was known at least since (26, 27). What
has been overlooked is that this discreteness is useful for
avoiding a problem with Jeffreys prior on the hyper-ribbon
parameter spaces generic in science (5): because it weights the
irrelevant parameter volume, Jeffreys prior has strong dependence on microscopic effects invisible to experiment. The limit
m → ∞ has erased the divide between relevant and irrelevant
parameters, by throwing away the natural length scale on the
parameter manifold. By contrast p? (θ) retains discreteness at
roughly this scale, allowing it to ignore irrelevant directions.
Along a relevant parameter direction this discreteness is no
worse than rounding θ to as many digits as we can hope to
measure, and we showed that in fact the spacing of atoms
decreases faster than our accuracy improves.
††

Edges of the parameter manifold give simpler models not only in the sense of
having fewer parameters, but also in an algorithmic sense. For example, the
Michaelis–Menten model is analytically solvable (53) in a limit which corresponds
to a manifold boundary (54). Stable linear dynamical systems of order n are model
∗∗
boundaries order n + 1 systems (55). Taking some parameter combinations to the
If we have more parameters than measurements then the model must be singular.
extreme can lock spins into Kadanoff blocks (54).
In fact the exponential model of Figure 4 is already slightly singular, since k1 ↔ k2
does not change the data; we could cure this by restricting to k2 ≥ k1 , or by working ‡‡ This view is natural in the objective Bayesian tradition, but see (58) and (59–61) for
with ~
y , to obtain a regular model.
alternative viewpoints.

Mattingly, Transtrum, Abbott, Machta

|

Rational Ignorance

PNAS

|

7 February, 2018

|

vol. XXX

|

no. XX

|

5

Model selection is more often studied not as part of prior
selection, but at the stage of fitting the parameters to data.
From noisy data, one is tempted to fit a model which is more
complicated than reality; avoiding such overfitting improves
predictions. The AIC, BIC (15, 62) and related criteria (19,
20, 63–65) are subleading terms of various measures in the
m → ∞ limit, in which all (nonsingular) parameters of the
true model can be accurately measured. Techniques like MDL,
NML, and cross-validation (63, 66, 67) need not take this limit,
but all are applied after seeing the data. They favor minimally
flexible models close to the data seen, while our procedure
favors one answer which can distinguish as many different
outcomes as possible. It is curious that both approaches can
point towards simplicity. We explore this contrast in more
detail in the Appendix.§§
Being discrete, the prior p? (θ) is very likely to exclude
the true value of the parameter, if such a θtrue ∈ Θ exists.
This is not a flaw: the spirit of effective theory is to focus on
what is relevant for describing the data, deliberately ignoring
microscopic effects which we know to exist (68). Thus the
same effective theory can emerge from different microscopic
physics [as in the universality of critical points describing phase
transitions (69)]. The relevant degrees of freedom are often
quasiparticles [such as the Cooper pairs of superconductivity
(70)] which do not exist in the microscopic theory, but give
a natural and simple description at the scale being observed.
We argued here for such simplicity not on the grounds of the
difficulty of simulating 1023 electrons, nor of human limitations,
but based on the natural measure of information learned.
There is similar simplicity to be found outside of physics.
For example the Michaelis–Menten law for enzyme kinetics
(71) is derived as a limit in which only the ratios of some reaction rates matter, and is useful regardless of the underlying
system. In more complicated systems which we cannot solve
by hand, and for which the symmetries and scaling arguments
used in physics cannot be applied, we hope that our information approach may be useful for identifying the appropriately
detailed theory.

6. M. K. Transtrum, B. B. Machta and J. P. Sethna, Why are Nonlinear Fits
to Data so Challenging?, Phys. Rev. Lett. 104 (2010) 060201
[arXiv:0909.3884].
7. M. K. Transtrum, B. B. Machta and J. P. Sethna, Geometry of nonlinear

least squares with applications to sloppy models and optimization,
Phys. Rev. E83 (2011) 036701 [arXiv:1010.1449].
8. B. B. Machta, R. Chachra, M. K. Transtrum and J. P. Sethna, Parameter
Space Compression Underlies Emergent Theories and Predictive
Models, Science 342 (2013) 604–607 [arXiv:1303.6738].
9. M. K. Transtrum, B. B. Machta, K. S. Brown, B. C. Daniels, C. R. Myers and J. P.
Sethna, Perspective: Sloppiness and emergent theories in physics,
biology, and beyond, J. Chem. Phys. 143 (2015) 010901
[arXiv:1501.07668].
10. T. O’Leary, A. C. Sutton and E. Marder, Computational models in the age
of large datasets, Cur. Op. Neurobio. 32 (2015) 87–94.
11. T. Niksic and D. Vretenar, Sloppy nuclear energy density functionals:
effective model reduction, Phys. Rev. C94 (2016) 024333
[arXiv:1606.08617].
12. D. V. Raman, J. Anderson and A. Papachristodoulou, Delineating
Parameter Unidentifiabilities in Complex Models, arXiv:1607.07705.
13. G. Bohner and G. Venkataraman, Identifiability, reducibility, and
adaptability in allosteric macromolecules, J. Gen. Physiol. 149 (2017)
547–560.
14. A. Raju, B. B. Machta and J. P. Sethna, Information geometry and the
renormalization group, arXiv:1710.05787.
15. H. Akaike, A new look at the statistical model identification, IEEE
Trans. Automat. Contr. 19 (1974) 716–723.
16. N. Sugiura, Further analysts of the data by Akaike’s information
criterion and the finite corrections, Comm. Stat. Th. Meth. 7 (1978)
13–26.
17. V. Balasubramanian, Statistical inference, Occam’s razor, and
statistical mechanics on the space of probability distributions,
Neural Comp. 9 (1997) 349–368.
18. I. J. Myung, V. Balasubramanian and M. A. Pitt, Counting probability
distributions: differential geometry and model selection., PNAS 97
(2000) 11170–11175.
19. D. J. Spiegelhalter, N. G. Best, B. P. Carlin and A. Van Der Linde, Bayesian
measures of model complexity and fit, J. Roy. Stat. Soc. B 64 (2002)
583–639.
20. S. Watanabe, Asymptotic equivalence of Bayes cross validation and

21.
22.
23.

ACKNOWLEDGMENTS.
We thank Vijay Balasubramanian,
William Bialek, Robert de Mello Koch, Peter Grünwald, Jon Machta,
James Sethna, Paul Wiggins, and Ned Wingreen for discussion and
comments. We thank ICTS Bangalore for hospitality.
H.H.M. was supported by NIH grant R01GM107103. M.K.T.
was support by NSF-EPCN 1710727. B.B.M. was supported by a
Lewis-Sigler Fellowship and by NSF PHY 0957573. M.C.A. was
supported by NCN grant 2012/06/A/ST2/00396.
1. L. P. Kadanoff, Scaling laws for Ising models near Tc , Physics 2 (1966)
263–272.
2. K. G. Wilson, Renormalization group and critical phenomena. 1.
Renormalization group and the Kadanoff scaling picture, Phys. Rev.
B4 (1971) 3174–3183.
3. J. L. Cardy, Scaling and renormalization in statistical physics. Cambridge Univ.
Press, 1996.
4. J. J. Waterfall, F. P. Casey, R. N. Gutenkunst, K. S. Brown, C. R. Myers, P. W.
Brouwer and J. P. Sethna, Sloppy-model universality class and the
Vandermonde matrix., Phys. Rev. Lett. 97 (2006) 150601–4.
5. R. N. Gutenkunst, J. J. Waterfall, F. P. Casey, K. S. Brown, C. R. Myers and J. P.
Sethna, Universally Sloppy Parameter Sensitivities in Systems
Biology Models, PLoS Comp. Biol. 3 (2007) e189–8.
§§

Model selection usually starts from a list of models to be compared, in our language
a list of submanifolds of Θ. We can also consider maximising mutual information in
this setting, rather than with an unconstrained function p(θ), and unsurprisingly we
observe a similar preference for highly flexible simpler models. This is also discussed
in the Appendix, at Eq. (S3).

PNAS

|

7 February, 2018

|

vol. XXX

|

no. XX

|

6

24.
25.

widely applicable information criterion in singular learning
theory, JMLR 11 (2010) 3571–3594 [arXiv:1004.2316].
P. A. Wiggins and C. H. LaMont, Information-based inference for
singular models and finite sample sizes, arXiv:1506.05855.
M. K. Transtrum and P. Qiu, Model reduction by manifold boundaries,
Phys. Rev. Lett. 113 (2014) 098701–6.
C. E. Shannon, A mathematical theory of communication, Bell Sys.
Tech. J. 27 (1948) 623–656.
D. V. Lindley, On a Measure of the Information Provided by an
Experiment, Ann. Math. Statist. 27 (1956) 986–1005.
A. Rényi, On some basic problems of statistics from the point of
view of information theory, Proc. 5th Berkeley Symp. on Math. Statist.

and Prob. (1967) 531–543. [projecteuclid.org].
26. G. Färber, Die Kanalkapazität allgemeiner Übertragunskanäle bei

begrenztem Signalwertbereich beliebigen Signalübertragungszeiten
sowie beliebiger Störung, Arch. Elektr. Übertr. 21 (1967) 565–574.
27. J. G. Smith, The information capacity of amplitude-and
variance-constrained scalar gaussian channels, Information and
Control 18 (1971) 203–219.
28. S. L. Fix, Rate distortion functions for squared error distortion
measures, Proc. 16th Annu. Allerton Conf. Commun. Control Comput. (1978)
704–711.
29. J. O. Berger, J. M. Bernardo and M. Mendoza, On Priors that Maximize
Expected Information, Recent Developments in Statistics and Their
Applications (1988) 1–20. [www.uv.es/~bernardo/].
30. Z. Zhang, Discrete Noninformative Priors. PhD thesis, Yale University, 1994.
[UMI 9523257].
31. J. M. Bernardo, Reference Posterior Distributions for Bayesian
Inference, J. Roy. Stat. Soc. B 41 (1979) 113–147. [www.uv.es/~bernardo/].
32. B. S. Clarke and A. R. Barron, Jeffreys’ prior is asymptotically least
favorable under entropy risk, J. Stat. Plan. Infer. 41 (1994) 37–60.
33. H. R. Scholl, Shannon optimal priors on independent identically

distributed statistical experiments converge weakly to Jeffreys’
prior, Test 7 (1998) 75–94.
34. H. Jeffreys, An Invariant Form for the Prior Probability in

Mattingly, Transtrum, Abbott, Machta

|

Rational Ignorance

Estimation Problems, Proc. Roy. Soc. A 186 (1946) 453–461.
35. J. E. Kerrich, An Experimental Introduction to the Theory of Probability.
Copenhagen: E Munksgaard, 1946.
36. C. O’Luanaigh, CERN data centre passes 100 petabytes, home.cern
(2013).
37. S. Arimoto, An algorithm for computing the capacity of arbitrary
discrete memoryless channels, IEEE Trans. Inform. Theory 18 (1972)
14–20.
38. R. Blahut, Computation of channel capacity and rate-distortion
functions, IEEE Trans. Inform. Theory 18 (1972) 460–473.
39. K. Rose, A mapping approach to rate-distortion computation and
analysis, IEEE Trans. Inform. Theory 40 (1994) 1939–1952.
40. D. Haussler, A general minimax result for relative entropy, IEEE
Trans. Inform. Theory 43 (1997) 1276–1280.
41. M. N. Ghosh, Uniform Approximation of Minimax Point Estimates,
Ann. Math. Statist. 35 (1964) 1031–1047.
42. G. Casella and W. E. Strawderman, Estimating a Bounded Normal
Mean, Ann. Statist. 9 (1981) 870–878.
43. I. Feldman, Constrained Minimax Estimation of the Mean of the
Normal Distribution with Known Variance, Ann. Statist. 19 (1991)
2259–2265.
44. M. Chen, D. Dey, P. Müller, D. Sun and K. Ye, Frontiers of statistical decision
making and Bayesian analysis. Springer, New York, NY, 2010.
45. C. A. Sims, Rational Inattention: Beyond the Linear-Quadratic
Case, American Economic Review 96 (2006) 158–163.
46. J. Jung, J. Kim, F. Matĕjka and C. A. Sims, Discrete Actions in
Information-Constrained Decision Problems, .
[princeton.edu/~sims/#RIDiscrete].
47. S. Laughlin, A simple coding procedure enhances a neuron’s
information capacity, Z. Naturforsch. 36 c (1981) 910–912.
48. G. Tkačik, C. G. Callan and W. Bialek, Information flow and
optimization in transcriptional regulation, PNAS 105 (2008)
12265–12270 [arXiv:0705.0313].
49. M. D. Petkova, G. Tkačik, W. Bialek, E. F. Wieschaus and T. Gregor, Optimal
decoding of information from a genetic network, arXiv:1612.08084.
50. A. Mayer, V. Balasubramanian, T. Mora and A. M. Walczak, How a
well-adapted immune system is organized, PNAS 112 (2015)
5950–5955 [arXiv:1407.6888].
51. M. C. Abbott and B. B. Machta, An information scaling law: ζ = 3/4,
arXiv:1710.09351.
52. S. H. Strogatz, Nonlinear Dynamics And Chaos. Sarat Book House, 2007.
53. S. Schnell and C. Mendoza, Closed Form Solution for
Time-dependent Enzyme Kinetics, J. Theor. Biol. 187 (1997) 207–212.
54. M. Transtrum, G. Hart and P. Qiu, Information topology identifies
emergent model classes, arXiv:1409.6203.
55. P. E. Paré, A. T. Wilson, M. K. Transtrum and S. C. Warnick, A unified view

model selection, Statistics Surveys 4 (2010) 40–79.
68. P. W. Anderson, More Is Different, Science 177 (1972) 393–396.
69. R. W. Batterman, Philosophical Implications of Kadanoff’s Work on
the Renormalization Group, J. Stat. Phys. 167 (2017) 559–574.
[pitt.edu/~rbatterm/].
70. J. Bardeen, L. N. Cooper and J. R. Schrieffer, Theory of
superconductivity, Phys. Rev. 108 (1957) 1175–1204.
71. L. Michaelis and M. L. Menten, The kinetics of invertin action, FEBS
Letters 587 (2013) 2712–2720. [Translation by T. R. C. Boyde].
72. R. E. Kass and A. E. Raftery, Bayes Factors, J. Amer. Stat. Assoc. 90 (1995)
773.
73. A. Bhadra, J. Datta, N. G. Polson and B. Willard, Default Bayesian
analysis with global-local shrinkage priors, Biometrika 103 (Dec.,
2016) 955–969.
74. D. Simpson, H. Rue, A. Riebler and T. G. Martins, Penalising Model
Component Complexity: A Principled, Practical Approach to
Constructing Priors, Statist. Sci. 32 (2017) 1–28 [arXiv:1403.4630].
75. J. I. Myung, D. J. Navarro and M. A. Pitt, Model selection by normalized
maximum likelihood, Journal of Mathematical Psychology 50 (2006)
167–179.
76. P. D. Grünwald, The Minimum Description Length Principle. MIT Press, 2007.
77. C.-I. Chang and L. D. Davisson, On calculating the capacity of an
infinite-input finite (infinite)-output channel, IEEE Trans. Inform.
Theory 34 (1988) 1004–1010.
78. J. Lafferty and L. Wasserman, Iterative Markov chain Monte Carlo
computation of reference priors and minimax risk, Proc. 17th conf.
Uncert. AI (2001) 293–300 [arXiv:1301.2286].
79. J. Dauwels, Numerical computation of the capacity of continuous
memoryless channels, Proc. 26th Symp. Inf. Th. Benelux (2005).
[www.dauwels.com].

Appendix: Model Selection from Data
The usual discussion of model selection takes place after observing
data x. If we wish to compare some models∗ labeled by d, each
with some prior pd (θ), then one prescription is to choose the model
with the largest p(x). Labelling this explicitly, we write
p(x|d) =

Mattingly, Transtrum, Abbott, Machta

|

Rational Ignorance

pd (θ) > 0 on Θd ⊂ Θ.

[S1]

If the Bayes factor p(x|d)/p(x|d0 ) is larger than one then (absent
any other information) d is preferred over d0 (72).† In the usual
asymptotic limit m → ∞, this idea leads to minimising the Bayesian
information criterion (BIC) (62):
− log p(x|d) ≈ − log p(x|θ̂d ) +

pp. 1989–1994, IEEE, 2015.
56. N. Lewis, Combining independent Bayesian posteriors into a

(2000) 1244–1255.
59. T. Seidenfeld, Why I am not an objective Bayesian; some
reflections prompted by Rosenkrantz, Theor Decis 11 (1979) 413–440.
60. R. E. Kass and L. Wasserman, The selection of prior distributions by
formal rules, J. Amer. Stat. Assoc. 91 (1996) 1343–1370.
61. J. Williamson, Objective Bayesianism, Bayesian conditionalisation
and voluntarism, Synthese 178 (2009) 67–85.
62. G. Schwarz, Estimating the dimension of a model, Ann. Statist. 6
(1978) 461–464.
63. J. Rissanen, Modeling by Shortest Data Description, Automatica 14
(1978) 465–471.
64. C. S. Wallace and D. M. Boulton, An Information Measure for
Classification, The Computer Journal 11 (1968) 185–194.
65. S. Watanabe, A widely applicable Bayesian information criterion,
JMLR 14 (2013) 867–897 [arXiv:1208.6338].
66. P. D. Grünwald, I. J. Myung and M. A. Pitt, Advances in Minimum Description
Length: Theory and Applications. 2009.
67. S. Arlot and A. Celisse, A survey of cross-validation procedures for

dθ p(x|θ) pd (θ),

Θd

of Balanced Truncation and Singular Perturbation
Approximations, in Proceedings of the American Control Conference,

confidence distribution, with application to estimating climate
sensitivity, J. Stat. Plan. Infer. (2017) to appear.
57. N. Lewis, Modification of Bayesian Updating where Continuous
Parameters have Differing Relationships with New and Existing
Data, arXiv:1308.2791.
58. D. Poole and A. E. Raftery, Inference for Deterministic Simulation
Models: The Bayesian Melding Approach, J. Amer. Stat. Assoc. 95

Z

where − log p(x|θ̂d ) =

1 2
χ
2

=

1
2

Pm
i=1

d
log m + O(m0 )
2
[xi − yi (θ̂d )]2 /σ 2 ∼ O(m),

and θ̂d is a maximum likelihood estimator for x, constrained to the
appropriate subspace:
θ̂d (x) = argmax p(x|θ).
θ∈Θd

The term d log m penalises models with more parameters, even
though they can usually fit the data more closely. Despite the name
this procedure is not very Bayesian: one chooses the effective model
(and hence the prior) after seeing the data, rather than simply
updating according to Bayes theorem.‡
∗

†

The word model unfortunately means several things in the literature. We mean
parameter space Θd always equipped with a likelihood function p(x|θ), and usually
with a prior pd (θ). When this is a subspace of some larger model ΘD (whose
likelihood function agrees, but whose prior may be unrelated) then we term the
smaller one an effective model, or a reduced model, although we do not always write
the adjective. The optimal prior p? (θ) defines an effective model in this sense. Its
support will typically be on several boundaries of ΘD . If the boundaries of ΘD (of
all dimensions) are regarded as a canonical list of reduced models, then p? (θ) is
seldom a sub-model of any one of them.
R
If one of the priors is improper, say

dθ pd (θ) = ∞, then p(x|d) will also be

infinite. In this sense the Bayes factor behaves worse than the posterior p(θ|x),
which can still be finite.
‡
Terms penalising more complex models can be translated into shrinkage priors, which
concentrate weight near to simpler models (73). Perhaps the shrinkage priors closest

PNAS

|

7 February, 2018

|

vol. XXX

|

no. XX

|

7

Related prescriptions can be derived from minimum description length (MDL) ideas. To allow reconstruction of the data we
transmit both the fitted parameters and the residual errors, and
minimising the (compressed) length of this transmission drives a
tradeoff between error and complexity (63, 64, 66). A convenient
version of this goes by the name of normalized maximum likeihood
(NML) (75, 76), and chooses the model d which maximizes
pNML
(x) =
d

Bayes Factor Comparison

A

1

0

0

0

1

x1

Full model,
pJ(𝜃)

0

x1

Optimal
p⭑(𝜃)

1

NML
1

x2

1

0

0

0

1

x1

Full model,
pJ(𝜃)

0

x1

Three edges,
separately

Zd =

,

Zd

Z



dx0 p x0 |θ̂d (x0 ) .

[S2]

Θd

• Figure S1A compares three models: the complete model (with
Jeffreys prior), the optimal model described by discrete prior
p? (~
y ), and an even simpler model with weight only on the
three vertices ~
y = (0, 0), ( 21 , 12 ), (1, 1).
• Figure S1B instead compares the complete model to three
different one-parameter models (along the three boundaries
of the allowed region of the ~
y plane) and a zero-parameter
model (one point ~
y , in no particularly special place). In terms
of decay rates the three lines are limits k1 = k2 , k1 = 0 and
k2 = ∞.

Corners
only

Bayes

B



This is not Bayesian in origin, and does not depend on the prior on
each effective model d, only its support Θd . The function pNML
(x)
d
is not expected data in the sense of p(x) — it is not the convolution
of the likelihood with any prior.§ In the asymptotic limit pNML
(x)
d
approaches p(x) from Jeffreys prior, and this criterion agrees with
BIC (63), but away from this limit they differ.
In Figure S1 we apply these two prescriptions to the exponential
example treated in the text. At each ~
x ∈ X we indicate which
of a list of models is preferred.¶ Figure S2 instead draws the
distributions being used.

NML Comparison

x2

1

p x|θ̂d (x)

1

One point

Fig. S1. Model selection from data point x. Each large panel here shows which
of a list of effective models is preferred after observing data x ∈ X . On the left the
criterion is maximizing Eq. (S1), on the right the criterion is Eq. (S2). We study the
same exponential model considered above, with σ = 1/10. Panel A compares our
optimal effective model with prior p? (θ) to the full model (with Jeffeys prior) and to an
even simpler model whose prior is just three delta functions. (These are drawn in the
legend below). Panel B compares the full model to three different one-dimensional
models, each allowing only one edge of Θ (with a uniform prior along this, i.e. the
one-dimensional Jeffreys prior) and also to a trivial model (just one point), again with
colors as indicated just below.

Different effective models are preferred for different values of data x.
At a given point x, if several models contain the same θ̂(x) then the
simplest among them is preferred, which in the NML case means
precisely the one with the smallest denominator Zd . In fact a trivial
model consisting of just one point Θ0 = θ̂(x) would always be
preferred if it were among those considered — there is no automatic
preference for models which can produce a wide range of possible
data.
By contrast our prior selection approach aims to be able to
distinguish as many possible outcomes in X as possible. Applied
to the same list of models as in Figure S1, this gives the following
fixed scores (base e):
Ifull = 1.296,
and

I? = 1.630,

Icorners = 1.098

Iupper = 0.852,
Ilower-left = 0.845,
Ilower-right = 1.418,
Ione-point = 0.

[S3]

to this paper’s are the penalised complexity priors of (74). Those are also reparameterization invariant, and also concentrate weight on a subspace of Θ, often a boundary.
However both the subspace (or base model) and the degree of concentration (scaling
parameter) are chosen by hand, rather than being deduced from p(x|θ).
§
This relevant optimization problem here can be described as minimizing worst-case
expected regret, written (75)
NML

pd

p(x|θ̂d (x))
,
q(x)

= argmin max log
x

q

θ̂d (x) ∈ Θd .

Perhaps the closest formulation of our maximum mutual information problem is that
our p? (x), the distribution on X not the prior, can be found as follows (40):

p? = argmin max
q∈B

θ

R
X

dx p(x|θ) log

p(x|θ)
q(x)

where q(x) is constrained to be a Bayes strategy i.e. to arise from some prior p? (θ).
Note the absence of θ̂d (x) and the presence of an integral over X , corresponding
to the fact thats that this maximization takes place without being given a subspace
Θd nor seeing data x. The resulting distributions on X are also different, as drawn in
Figure S2. If plotted on Figure 5B, pNML
(x) from the full model would be somewhere
2
between the two expected data p(x) lines there. But it is not a comparable object; its
purpose is model comparison as in Figure S1.
¶
Recall that ~
x is ~
y corrupted by Gaussian noise, and ~
y is constrained to the area show
in Figure 4A because it arises from decay rates kµ via Eq. (5). We may regard
either yt or kµ as being the parameters, generically θ .

PNAS

|

7 February, 2018

|

vol. XXX

|

no. XX

|

8

Mattingly, Transtrum, Abbott, Machta

|

Rational Ignorance

B

A

pNML
(x)
d

x2

p(x|d)

■

■
■

■

�

■

x1
�

x

x2

pNML
(x)
d

■

�

d=

x

Full model, pJ(𝜃)
Optimal p⭑(𝜃)
Corners only

x1
Fig. S2. Distributions over X . Panel A shows p(x|d) Eq. (S1) and pNML
(x)
d
Eq. (S2) for the one-parameter Gaussian example Eq. (2) with σ = 1/10. The
three models being compared are the full model (with a flat prior), the simpler model
defined by p? (θ), and a model with just the endpoints of the line. Under the Bayes
factor comparison, the p? (θ) model would never be preferred here. Panel B draws
pNML
(x) for the two-parameter exponential model, Eq. (5), for the complete model,
d
the effective model defined by the support of p? (θ), and an even simpler model
allowing only three points — the same three models as compared in Figure S1A.
Notice that pNML
(x) is always a constant on the allowed region x ∈ y(Θd ).
d

By definition p? (θ) has the highest score. In second place comes the
line along the lower edge (corresponding to k1 = k2 ). The shorter
lines are strongly disfavored, because they cover a much smaller
range of possible data.

Algorithms
The standard algorithm for maximizing channel capacity (of discrete
memoryless channels) was written down independently by Blahut
(38) and Arimoto (37). This aspect of rate-distortion theory is
mathematically the same as the problem we consider, of maximizing
mutual information by choosing the prior. The algorithm starts
with p0 (θ) = const., and then at each time step updates this by
pτ +1 (θ) =

1 fKL (θ)
e
pτ (θ)
Zτ

0

[S4]

where Zτ =
dθ0 efKL (θ ) pτ (θ0 ) maintains normalization, and
fKL (θ) = DKL [p(x|θ)kp(x)] is computed with pτ (θ). Since this

R

𝜏 = 106 iterations

�

x2

■

■

𝜏 = 1000

Fig. S3. Convergence of the BA algorithm (S4) for the exponential model. This
shows p? (~
y ) for the case σ = 1/50, with ~
y discretized on a grid of spacing 1/100
in the bulk and 1/200 along the boundaries of the allowed region.

x1

■
■

𝜏 = 100

Mattingly, Transtrum, Abbott, Machta

|

Rational Ignorance

is a convex optimization problem, the algorithm is guaranteed to
converge to the global maximum. This makes it a good tool to see
discreteness emerging.
Figures 2 and S3 show the progress of this algorithm for the
one- and two-dimensional models in the text. We stress that the
number and positions of the peaks which form are unchanged when
the discretization of θ is made much finer. Notice also that the
convergence to delta functions happens much sooner near to the
boundaries than in the interior. The convergence to the correct
mutual information I(X; Θ), and towards the optimum distribution
on data space p(x), happens much faster than the convergence to
the correct number of delta functions.
Because θ must be discretized for this procedure, it is poorly
suited to high-dimensional parameter spaces. However once we
know that p? (θ) is discrete it is natural to consider algorithms
exploiting this. With K atoms, we can adjust their positions θ~a
and weights λa using gradients
∂MI
= λa
∂θaµ

Z
dx

~
~
p(x|θ)
∂p(x|θ)
log
µ
∂θ
p(x)

~ θ
~a
θ=

∂MI
= fKL (θ~a ) − 1.
∂λa

[S5]

Figures 1, 2A, 5 and the square plot points in Figure 4 were generated this way. This optimization is not a convex problem (there is
some tendency to place two atoms on top of each other, and thus
use too few points of support) but it can often find the optimum
solution. We can confirm this by calculating fKL (θ) everywhere
— any points for which this is larger than its value at the atoms
indicates that we do not have the optimal solution, and should add
an atom.
Monte Carlo algorithms for this problem have been investigated
in the literature, see (77, 78) and especially (79). (Incidentally, we
observe that (77)’s table 1 contains a version of scaling law Eq. (4),
with ζ ≈ 1/2. No attempt was made there to use the optimal
number of atoms, only to calculate the channel capacity to sufficient
accuracy.)

PNAS

|

7 February, 2018

|

vol. XXX

|

no. XX

|

9

