arXiv:1701.08895v2 [] 22 May 2017

Chentsov’s theorem for exponential families
James G. Dowty
May 23, 2017

Abstract
Chentsov’s theorem characterizes the Fisher information metric on statistical models
as essentially the only Riemannian metric that is invariant under sufficient statistics. This
implies that each statistical model is naturally equipped with a geometry, so Chentsov’s
theorem explains why many statistical properties can be described in geometric terms.
However, despite being one of the foundational theorems of statistics, Chentsov’s theorem has only been proved previously in very restricted settings or under relatively strong
regularity and invariance assumptions. We therefore prove a version of this theorem for
the important case of exponential families. In particular, we characterise the Fisher information metric as the only Riemannian metric (up to rescaling) on an exponential family
and its derived families that is invariant under independent and identically distributed
extensions and canonical sufficient statistics. Our approach is based on the central limit
theorem, so it gives a unified proof for both discrete and continuous exponential families,
and it is less technical than previous approaches.

1

Introduction

Chentsov’s theorem is a foundational theorem in statistics that characterizes the Fisher
information metric on statistical models as the only Riemannian metric (up to rescaling)
that is invariant under certain, statistically important transformations [10, 16, 9, 2, 7].
This effectively means that the Fisher information metric is the only natural metric on
a statistical model, so many statistical properties of these models should be describable
in terms of this metric. Known examples of this correspondence between statistical and
geometric properties include: the Cramér-Rao lower bound for the variance of an unbiased
estimator in terms of the inverse of the Fisher information metric [1, Thm. 2.2]; orthogonality as a criterion for first-order efficiency of estimators [1, Thm. 4.3]; the central role
of statistical curvature in the information loss of an efficient estimator [12, §3.3] and in
second-order efficiency [12, §3.4]; and the spontaneous emergence of the Fisher information volume [15] in the minimum description length (MDL) approach to statistical model
selection [6].
The original version of Chentsov’s theorem [10, 16, 9] only applied in the restricted
setting of statistical models with finite data spaces. This version of the theorem says that
the Fisher information metric is the only metric (up to a multiplicative constant) that is
defined on all models with finite data spaces and is invariant under all sufficient statistics.
Recall that a statistical model M is a (sufficiently regular) set of probability measures on
the same measurable space X , which we call the data space of M, and that a sufficient
statistic for M is a function on X for which the conditional distribution of any measure

1

P in M, given the sufficient statistic, is the same for all P . Sufficient statistics induce
corresponding maps on statistical models (the measure-theoretic push-forward maps) and
the invariance assumption above is that all of these maps are isometries (i.e., distancepreserving maps).
Since the assumption of finite data spaces is very restrictive, Ay et al. [2] proved a
version of Chentsov’s theorem that applies to models whose data spaces X are smooth
manifolds. Their version says that the Fisher information metric is the only metric (up to
rescaling) that is defined on all statistical models with a given data space X and is invariant
under all sufficient statistics, including discontinuous ones. This version of Chentsov’s theorem applies to many interesting statistical models but it makes very strong assumptions
about both the breadth of the models on which the metrics are defined and the invariance
properties of these metrics. Therefore Bauer et al. [7] proved a version of Chentsov’s theorem which says the Fisher information metric is the only metric (up to rescaling) that
firstly is defined on the space of all smooth, positive densities on a compact manifold X
of dimension 2 or higher and secondly is invariant under all diffeomorphisms from X to
itself (where diffeomorphisms are smooth maps with smooth inverses, so they are a special
type of sufficient statistic). The proof of Bauer et al. [7] was based on results from the
theory of generalized functions, especially the Schwartz kernel theorem [11, §6.1], and it
made far weaker invariance assumptions than that of Ay et al. [2]. The assumption that X
is a compact manifold without boundary excludes many cases of interest to statisticians,
though Bauer et al. [7] say this assumption can be weakened.
Despite their beauty and generality, the results of Ay et al. [2] and Bauer et al. [7]
leave open the possibility that there might exist a natural metric other than the Fisher
information metric on an individual statistical model M. This could occur, for example,
if there is a natural metric on M that does not (invariantly) extend to a metric on the
infinite-dimensional models of [2] and [7] that contain M and many unrelated models.
Also, exponential families have a distinguished, finite-dimensional set of sufficient statistics,
called the canonical sufficient statistics, which are related to their natural affine structures
([1, Thm. 2.4] and [3, Lemma 8.1]). Therefore, the invariance assumptions of [2] and [7]
are arguably too strong for exponential families, and instead it would be more natural to
consider invariance under canonical sufficient statistics rather than all sufficient statistics.
In this paper, we prove a refined version of Chentsov’s theorem in the important case
of exponential families. Instead of considering metrics defined on an infinite-dimensional
statistical model, as in [2] and [7], we consider metrics defined only on a given exponential
family M and some of its derived families, namely its independent and identically distributed (IID) extensions and their corresponding natural exponential families. Instead of
assuming these metrics are invariant under all sufficient statistics or all diffeomorphisms,
we assume invariance under canonical sufficient statistics and IID extensions. This assumption of invariance under IID extensions has no analogue in previous work, but IID
extensions are natural and important transformations between statistical models (perhaps
more so than sufficient statistics), so this invariance assumption is arguably more natural
than invariance under sufficient statistics. Also, this extra invariance assumption is offset
by the fact that we restrict our sufficient statistics to the canonical ones. Then, under
a mild regularity condition, we prove that metrics with these invariance properties are
multiples of the Fisher information metric (see Theorem 1 in Section 5). This result therefore gives a new characterisation of the Fisher information metric as the only metric on
an exponential family and its derived families that is invariant under canonical sufficient
statistics and IID extensions.
Our approach has a number of advantages: as discussed above, we only assume that
the metric is defined on an individual model and its related models, and our invariance
assumptions respect the natural affine structures of exponential families; we only consider metrics on a collection of finite-dimensional models (similar to the original version
of Chentsov’s theorem [10, 16, 9]), which allows us to avoid the technicalities encountered
in [2] and [7] because of the infinite-dimensionality of their statistical models; our proof

2

is unified for discrete and continuous distributions, unlike the proofs of [10, 16, 9] and
[7], so there is some hope of extending our proof to general statistical models; our proof
shows that Chentsov’s theorem is a corollary of the central limit theorem, which makes this
result more understandable and intuitive; and our results complement those of [7], since
(curved) exponential families are essentially the only statistical models with smooth sufficient statistics that are not diffeomorphisms, by the Pitman–Koopman–Darmois theorem
[5].
The rest of this paper is set out as follows. In Section 2 we define the Fisher information
metric and some relevant notions from differential geometry, as they apply in our main case
of interest. In Section 3 we briefly recall the definition of an exponential family and some
of its derived families. We then give precise descriptions of our assumptions in Section 4,
before using these assumptions and the central limit theorem to prove our characterisation
of the Fisher information metric in Section 5. We then describe an extension of our proof
to higher-order symmetric tensors in Section 6, before finishing with a discussion of our
results in Section 7. Section 7 also begins with a non-technical summary of our proof.

2

The Fisher information metric

This section briefly recalls the definitions of tangent vectors and the Fisher information
metric of a statistical model. A general reference for the notions from Riemannian geometry described here is [12, Appendix C].
In all later sections of this paper, we will take M to be a regular exponential family
with natural parameter space Θ, but in this section we let M be a more general statistical
model and let Θ be any smooth parameter space for M. More precisely, suppose Θ is an
open subset of Rd and that µ is a measure on Rm with support X . Then our statistical
model is M = {pθ µ | θ ∈ Θ}, where each pθR: X → R>0 is a µ-integrable, strictly positive
function that is normalized, meaning 1 = pθ dµ. Note that M is a set of probability
measures on Rm . We assume that the parameterisation of M by Θ is smooth, in the sense
that θ 7→ pθ (x) is a smooth (i.e., infinitely differentiable) function for µ-almost all x. We
also assume that the parameterisation is non-singular, meaning that the parameterisation
map Θ → M given by θ 7→ pθ µ is injective and that it maps non-zero tangent vectors to
non-zero tangent vectors, in a sense which will become clear below.
Because Θ is an open subset of Rd , any tangent vector u to Θ is a pair u = (θ, a)
for some θ ∈ Θ and some a ∈ Rd , where θ is called the base-point of u. The set of
all such tangent vectors, which is denoted T Θ and is called the tangent bundle of Θ, is
therefore T Θ = Θ × Rd . The tangent bundle is not a vector space in general, but the
set of all tangent vectors with the same base-point is. The vector space Tθ Θ consisting of
all vectors with base-point θ is called the tangent space to Θ at θ. Addition and scalar
multiplication in this vector space are given by
su + tv = (θ, sa + tb)

(1)

for any u, v ∈ Tθ Θ and any s, t ∈ R, where u = (θ, a) and v = (θ, b). Note that addition
and scalar multiplication in Tθ Θ effectively ignore the shared base-point θ.
Similarly, we can view each tangent vector to the statistical model M as a pair (P, A),
where the base-point P is an element of the model M and A is essentially the score in
a particular direction [14, §3.3]. More precisely, for each tangent vector u = (θ, a) to Θ,
there is a corresponding tangent vector ũ = (P, A) to M given by
P = pθ µ and A =

d
X
i=1

ai

∂pθ
µ.
∂θi

(2)

(The function taking u to ũ is the differential of the parameterisation θ 7→ pθ µ [12, Def.
C.3.4].) Let the tangent bundle T M of M be the set of all such tangent vectors, i.e., let

3

T M = {ũ | u ∈ T Θ}. Also, let the tangent space TP M to M at P ∈ M be the vector
space consisting of all tangent vectors (P, A) ∈ T M with base-point P . Even though
we have used a particular parameterisation of M to define TP M, this tangent space is
natural, in the sense that TP M is the same for all smooth parameterisations.
The Fisher information metric g F on M is given by
Z
dA dB
dP
(3)
g F (ũ, ṽ) =
dP dP
for any tangent vectors ũ = (P, A) and ṽ = (P, B) in the tangent space TP M [7, §3],
where dA/dP and dB/dP are Radon-Nikodym derivatives [8, §3.2]. It is straightforward
(see Appendix A) to show that definition (3) for the Fisher information metric reduces to
the usual, parameterisation-dependent definition [1, eq. 2.6]. However, the formulation
(3) will be more useful to us than the usual definition. Also, because (3) is phrased only
in terms of natural constructions, this formula makes it clear that g F does not depend on
arbitrary choices, such as the choice of parameterisation.
A Riemannian metric on a set is just a function that puts an inner product on each
of the set’s tangent spaces (if the set is suitably regular and the inner products vary
smoothly with the base-point). For example, a Riemannian metric on Θ can be thought
of as a smooth, matrix-valued function on Θ whose value at θ ∈ Θ is a d × d, symmetric,
positive definite matrix ḡθ , since this defines an inner product on each Tθ Θ with the inner
product of any u, v ∈ Tθ Θ being g(u, v) = aT ḡθ b, where u = (θ, a) and v = (θ, b).
In our main case of interest, where M is an exponential family, the integral in (3)
always converges [12, Thm. 2.2.5]. Then it is not hard to see that (3) defines an inner
product on each tangent space to M (and this varies smoothly with the base-point), so
the Fisher information metric g F is a Riemannian metric on M.

3

Exponential families and their derived families

Partly to establish our notation, this section briefly recalls the definitions of an exponential
family, its IID extensions and their corresponding natural exponential families.

3.1

Exponential families

Let µ be a measure on Rm and let T : X → Rd be a measurable function, where X ⊆ Rm
is the support of µ. Let


Z
d
Θ= θ∈R
exp(θ · T )dµ < ∞ ,
where the dot (·) denotes the Euclidean inner product on Rd . For each θ ∈ Θ, define
pθ : X → R>0 by
pθ (x) = exp(θ · T (x))/Z(θ)

(4)
R

for any x ∈ X , where Z : Θ → R is the partition function Z(θ) = exp(θ · T )dµ.
Assume that Θ is a non-empty, open subset of Rd and that T is full rank, in the sense
that the image of T is not contained in any (d − 1)-dimensional hyperplane in Rd . Then
M = {pθ µ | θ ∈ Θ} is a regular exponential family of order d with dominating measure
µ and canonical sufficient statistic T , and all regular exponential families are of this form
[3, §8.1]. Note that each element of M is a probability measure on Rm .

3.2

IID extensions

The n-fold IID extension Mn of M is the set Mn = {P n | P ∈ M} of all measures
of the form P n for some P ∈ M, where P n = P × · · · × P (with n copies of P ) is the

4

product measure on X n [8, §3.3]. In terms of the parameterisation (4), Mn is the set
(n)
(n)
of all measures of the form pθ µn for some θ ∈ Θ, where pθ : X n → R>0 is given by
(n)
pθ (x1 , . . . , xn ) = pθ (x1 ) . . . pθ (xn ) and µn = µ × · · · × µ is the product measure on X n
[3, Example 8.12(ii)]. So by (4),
(n)

pθ

= exp(nθ · Tn − n log Z(θ)),

(5)

where Tn : X n → Rd is given by Tn (x1 , . . . , xn ) = (T (x1 ) + · · · + T (xn ))/n for any
x1 , . . . , xn ∈ X . Therefore Mn is an exponential family with dominating measure µn
and sufficient statistic Tn (and natural parameter nθ, see [12, Thm. 2.2.6]). Note that
(1)
M1 = M, T1 = T and pθ = pθ .

3.3

Natural exponential families

Recall that if Y and Z are measurable spaces, φ : Y → Z is a measurable function and P
is a measure on Y then the push-forward of P via φ is the measure φ∗ P on Z given by
(φ∗ P )(U ) = P (φ−1 (U ))

(6)

for any measurable set U in Z [8, §3.6]. This immediately implies that if Y is a Yvalued random variable with distribution P then φ(Y ) is a Z-valued random variable with
distribution φ∗ P , which in symbols we write as
Y ∼ P implies φ(Y ) ∼ φ∗ P.

(7)

Then the natural exponential family corresponding to Mn and Tn is the set Nn =
{Tn ∗ P n | P n ∈ Mn } of measures on Rd . By [3, Examples 8.12(ii) and 8.12(iii)], Nn =
{qθn νn | θ ∈ Θ}, where νn is a measure on Rd which does not depend on θ and qθn : Rd →
R>0 is given by
qθn (y) = exp(nθ · y − n log Z(θ))

(8)

for any y ∈ Rd . The formula (8) shows that the superscript in qθn is actually an exponent,
so we will write qθ for qθ1 (and then the notation qθn is unambiguous).
Note that even though M, M2 , M3 , . . . and N1 , N2 , N3 , . . . are families of measures
on different spaces (namely, X , X 2 , X 3 , . . . and Rd , Rd , Rd , . . . , respectively), they are all
parameterised by Θ ⊆ Rd so they are all d-dimensional families of measures.

4

Invariance and regularity conditions

Let M, Mn and Nn be as in Section 3 and suppose now that these spaces have been
equipped with Riemannian metrics g, g n and gn , respectively. In this section, we will give
precise conditions that formalize the notion of these metrics being invariant under IID
extensions and canonical sufficient statistics, as well as giving a mild regularity condition.
These conditions will then be used in Section 5 to prove our main theorem. See Section
4.4 for a number of remarks about these assumptions.
Assumptions. We make the following assumptions, which are described precisely in the
subsections below:
A1 The metrics g and g n are invariant under IID extensions (up to a factor of n)
A2 The metrics g n and gn are invariant under canonical sufficient statistics
A3 The norms corresponding to the metrics gn can all be calculated by a function that
satisfies a weak continuity condition

5

4.1

A1: Invariance under IID extensions

Let IIDn : M → Mn be the function which maps each P ∈ M to the product measure
P n = P × · · · × P (see Section 3.2). Then our first assumption is that this map is an
isometry (i.e., distance-preserving map) up to a factor of n.
More precisely, let u = (θ, a) ∈ T Θ be any tangent vector to Θ, as in Section 2. Then
similarly to (2), u corresponds under the smooth parameterisation (5) to a tangent vector
Pd
(n)
(n)
ũn to Mn , where ũn = (P n , A(n) ), P n = pθ µn and A(n) = i=1 ai (∂pθ /∂θi )µn . Let
n
n
n
T M = {ũ | u ∈ T Θ} be the set of all such tangent vectors to M . Then our first
assumption is that
g n (ũn , ṽ n ) = ng(ũ, ṽ)

(9)

for all tangent vectors u, v ∈ T Θ with the same base-point. Here, ṽ and ṽ n are the tangent
vectors to M and Mn (respectively) corresponding to v ∈ T Θ, as for u above. Note that
(9) just says that g n = ng under the identification of M with Mn via IIDn .
The Fisher information metric is invariant under IID extensions in the sense of (9) by
[1, eq. 4.2], so assumptions (A1)–(A3) cannot characterize the Fisher information metric
unless the factor of n is included in (9) (though see Remark 4).

4.2

A2: Invariance under canonical sufficient statistics

Let Tn : X n → Rd be the canonical sufficient statistic from Section 3.2 and let Tn ∗ :
Mn → Nn be the corresponding (measure-theoretic) push-forward map of Tn , see Section
3.3. Then our second assumption is that this map Tn∗ is an isometry (and that all other
canonical sufficient statistics are isometries, in a sense which will be made precise in Section
4.3).
More precisely, let u = (θ, a) ∈ T Θ be any tangent vector to Θ, as in Section 2. Then
similarly to (2), u corresponds under the smooth parameterisation (8) to a tangent vector
ũn = (Qn , An ) to Nn , where
Qn = qθn νn and An =

d
X

ai (∂qθn /∂θi )νn .

(10)

i=1

Let T Nn = {ũn | u ∈ T Θ} be the set of all such tangent vectors. Then our second
assumption is that
gn (ũn , ṽn ) = g n (ũn , ṽ n )

(11)

for all tangent vectors u, v ∈ T Θ with the same base-point. Here, ṽ n and ṽn are the
tangent vectors to Mn and Nn (respectively) corresponding to v ∈ T Θ, as for u above.
Note that (11) just says that gn = g n under the identification of Mn with Nn via Tn ∗ .

4.3 A3: Calculability of norms by a function that satisfies a weak
continuity condition
p
Let h be the norm corresponding to g, so h(ũ) = g(ũ, ũ) for any ũ ∈ T M. Note that h
determines g by the polarisation formula,


g(ũ, ṽ) = h2 (ũ + ṽ) − h2 (ũ − ṽ) /4

for any ũ, ṽ ∈ T M with the same base-point (which follows from the bilinearity of g), so
any question about g can be phrased in terms of h. However, it will be more convenient
to work with h than g, because h is a function defined on T M, whereas g is only defined
on certain pairs of tangent vectors (those with
p the same base-point). Similarly, let hn be
the norm corresponding to gn , so hn (ũn ) = gn (ũn , ũn ) for any ũn ∈ T Nn .

6

Let T ′ be the set of all pairs (P, A), where P is a probability measure on Rd and A
is a signed measure on Rd , and note that T Nn ⊆ T ′ for every n. Then our regularity
condition (A3) is, firstly, that there is subset T of T ′ and a function H : T → R so that,
for each n, T Nn ⊆ T (i.e. H is defined on each T Nn ) and
hn (ũn ) = H(ũn )

(12)

for every ũn ∈ T Nn . In other words, we assume that there is some function H whose
restriction to each T Nn is the norm hn . For instance, we could take T = ∪∞
n=1 T Nn and
then define H by the requirement that (12) holds, which gives a well-defined H whenever
the functions hn agree on any overlaps between the spaces T Nn .
Further, we assume that H has the following weak continuity property. Firstly, we
require that H is defined on all pairs of the form (Φ, f Φ), where Φ is the probability
measure for the standard normal distribution on Rd and f : Rd → R is a linear function
(with f (0) = 0). Secondly, we require that
H(Pn , f Pn ) = H(Φ, f Φ)

(13)

for any sequence Pn of probability measures on Rd for which H(Pn , f Pn ) is constant in n,
Pn ⇒ Φ and each Pn is standardized (i.e., Pn has 0 mean and identity variance-convariance
matrix), where H(Pn , f Pn ) is the value of the function H at (Pn , f Pn ) ∈ T and Pn ⇒ Φ
means Pn converges to Φ in the sense of the weak convergence of measures [13, Def. 1.2.1].
This condition is an extremely weak form of continuity, see Remark 1.
Lastly, as a consequence of our assumption (A2) that the metrics should be invariant
under all canonical sufficient statistics, we assume that H is affine invariant (see Remark
6). Here, an invertible affine transformation of Rd is a map L : Rd → Rd of the form
L(x) = M x + c for some invertible d × d matrix M and some c ∈ Rd . The push-forward
L∗ A of any signed measure A on Rd is defined in a similar way to the push-forward of an
(unsigned) measure, see (6). We define the push-forward L∗∗ (P, A) of any (P, A) ∈ T to
be L∗∗ (P, A) = (L∗ P, L∗ A). (In this notation, L∗ is the measure-theoretic push-forward,
which is a map from the space of signed measures on Rd to itself, and L∗∗ is the differential
of this map if (P, A) is interpreted as a tangent vector.) Then our condition that H is
affine invariant means that L∗∗ (P, A) ∈ T and
H(L∗∗ (P, A) = H(P, A)

(14)

for every (P, A) ∈ T and every invertible affine transformation L of Rd .
For future reference, we note that if L is an invertible affine transformation, P is a
probability measure and f is a P -integrable, real-valued function then
L∗ (f P ) = (f ◦ L−1 )L∗ P

(15)

by the change of variables formula [8, Thm. 3.6.1].

4.4

Remarks on the assumptions

Remark 1. Assumptions (A1) and (A2) say that the metrics on M, Mn and Nn are
invariant under a countable set of transformations and, in a certain sense, under the
finite-dimensional group of affine transformations of Rd . The third assumption (A3) is an
extremely weak form of continuity. Firstly, this condition says that the norms hn agree on
any overlaps between the spaces T Nn , so that these functions can be pieced together into a
single function H. Secondly, this condition says that if f is linear and Pn ⇒ Φ is a sequence
for which (Pn , f Pn ) all have the same norms then this shared norm must be H(Φ, f Φ). By
comparison, full continuity of H would require that limn→∞ H(Pn , fn Pn ) = H(P, f P ) for
every sequence (Pn , fn Pn ) in T that converges to (P, f P ) (with respect to some notion of
convergence). So our third assumption is the condition for the continuity of H in the very
special case where P = Φ, H(Pn , fn Pn ) is constant in n, fn = f for every n and f is a
linear function.

7

Remark 2. Recent versions of Chentsov’s theorem [2, 7] consider metrics on infinitedimensional statistical models that are invariant under infinite-dimensional sets of transformations. This infinite dimensionality introduces technical complications and it makes
strong assumptions about both the space on which the metric is defined and its symmetries. By contrast, our approach allows us to only consider metrics on a collection of
finite-dimensional models, as in the original version of Chentsov’s theorem [10, 16, 9].
This allows our characterisation of the Fisher information metric to be relatively free from
technicalities and it allows us to make relatively weak invariance and regularity assumptions.
Remark 3. It is not hard to see that the Fisher information metric satisfies assumptions
(A1)–(A3). For it is well known that the Fisher information metric is invariant under both
IID extensions (in the sense of (9)) and sufficient statistics [1, eq. 4.2 and Thm. 2.1].
Also, given any probability measure P on Rd , let TP = {(P, f P ) | f ∈ L2 (Rd , P )}, and let
T be the union of these spaces TP as P ranges over the set of all probability measures on
Rd . Then by (3), the Fisher information norm H F (P, f P ) of any (P, f P ) ∈ T is just the
L2 (Rd , P )-norm of f . So if f is a linear function on Rd , say f (y) = c · y for some c ∈ Rd ,
and Q is any standardized probability measure on Rd then
s Z
sZ

√
F
2
yy T dQ(y) c = cT Ic = kck,
(c · y) dQ(y) = cT
H (Q, f Q) =
where kck is the Euclidean norm of c ∈ Rd . So for any sequence Pn of standardized
probability measures (whether weakly convergent to Φ or not), H F (Pn , f Pn ) = kck =
H F (Φ, f Φ), so H F satisfies the weak continuity condition (13). Lastly, this function H F
is affine invariant (14) by the change of variables formula (15).
Remark 4. In some ways the factor of n in (9) is not essential, since we could instead
formulate our assumptions and theorems in terms of the metrics ġ n = g n /n and ġn = gn /n,
in which case (9) would be equivalent to the equation that describes exact invariance under
the map IIDn , rather than invariance up to a factor of n (though H as in (12) might
not exist without the factor of n). However, it is natural to include the factor of n in
our formulation of IID invariance, firstly because the Fisher information metric is IID
invariant in the sense of (9) [1, eq. 4.2], so assumptions (A1)–(A3) would not characterise
the Fisher information metric without this factor, and secondly because the factor of n
arises from a natural construction from differential geometry (see Remark 5).
Remark 5. Given an arbitrary Riemannian metric g on M, a natural construction from
differential geometry gives a metric on the n-fold IID extension
Mn of M equal to the
Qn
M of M with itself n times
metric g n satisfying (9), as follows. The Cartesian product
is the space whose points are n-tuples (P1 , . . . , Pn ) of measures P1 , . . . , Pn ∈ M on X .
Given such an n-tuple, there is a corresponding product measure P1 × · · · × Pn on X n , and
conversely we can recover each Pi from P1 × · · · × Pn by marginalizing, so we can identify
(P1 , . . . , Pn ) with the product measure P1 × · · · × Pn on X n . This product measure is the
joint distribution of independent random variables X1Q
, . . . , Xn whose marginal distributions
are P1 , . . . , Pn , respectively. So if (P1 , . . . , Pn ) ∈ n M satisfies P1 = · · · = Pn then
P1 × · · · × Pn is the joint distribution of IID random variables X1 , . . . , Xn . Therefore we
can identify the diagonal
)
(
n
Y
M P1 = · · · = Pn
∆ = (P1 , . . . , Pn ) ∈
Qn
of
M with the n-fold IID extension Mn of M.
QnBut a Riemannian metric on M induces
M, and then ∆ inherits a metric from
a Riemannian metric
on
the
Cartesian
product
Q
its super-manifold n M. Under the above identification between ∆ and Mn , this metric
is the metric g n on Mn that satisfies (9).

8

Remark 6. The canonical sufficient statistics for an exponential family are only unique
up to affine transformations [3, Lemma 8.1], meaning that if L is an invertible affine
transformation of Rd and Tn : X n → Rd is a canonical sufficient statistic then L ◦ Tn is
also a canonical sufficient statistic (and every canonical sufficient statistic is of this form).
Replacing Tn by L ◦ Tn effectively replaces each tangent vector ũn ∈ T Nn by L∗∗ ũn , so
(11), (12) and the analogous equations for L ◦ Tn imply H(L∗∗ ũn ) = H(ũn ) for every
ũn ∈ T Nn . So since L is arbitrary, H is affine invariant.

5

The main theorem

We can now prove our version of Chentsov’s theorem. This theorem characterises the
Fisher information metric as the only metric (up to rescaling) on an exponential family
that is invariant under IID extensions and canonical sufficient statistics.
Let g F , g nF and gnF be the Fisher information metrics on M, Mn and Nn , respectively.
Theorem 1. Suppose that assumptions (A1)–(A3) of Section 4 hold. Then there is some
c > 0 so that g = cg F , g n = cg nF and gn = cgnF for every integer n ≥ 1.
Proof. Let any integer n ≥ 1 and any θ ∈ Θ be given, and let Q1 = qθ ν1 ∈ N1 and
Qn = qθn νn ∈ Nn be the corresponding distributions in N1 and Nn . By Theorem 2.2.6
of [12] and the comments preceding it, if Y1 , . . . , Yn are independent random variables all
distributed according to Q1 then their mean is distributed as Qn , which we write as
(Y1 + · · · + Yn )/n ∼ Qn .

(16)

Alternatively, it is not hard to prove (16), since if X1 , . . . , Xn ∼ pθ µ are IID and Yi′ =
T (Xi ) then Y1′ , . . . , Yn′ ∼ Q1 are IID and (Y1′ + · · · + Yn′ )/n = Tn (X1 , . . . , Xn ) ∼ Qn , by (7)
and since Q1 = T∗ P and Qn = Tn∗ P n (by definition), where P = pθ µ. This proves (16)
because Y1 , . . . , Yn and Y1′ , . . . , Yn′ have the same joint distribution so their means have
the same distribution, by another application of (7).
By (16), the mean τθ for Q1 is the same as that for Qn , i.e.
Z
Z
τθ = ydQ1 (y) = ydQn (y),
(17)
and the variance-covariance matrix Σθ for Q1 is n times that for Qn , i.e.
Z
Z
Σθ = (y − τθ )(y − τθ )T dQ1 (y) = n (y − τθ )(y − τθ )T dQn (y).

(18)

Now, let u = (θ, a) ∈ Tθ Θ be any tangent vector to Θ at θ, and define f : Rd → R by
1/2
1/2
f (y) = (Σθ a)·y for any y ∈ Rd . Here, Σθ is defined in the standard way via a diagonalisation of the symmetric, positive-definite matrix Σθ . As before, let ũ and ũn , respectively,
be the tangents to M and Nn that correspond to u under the parameterisations (5) and
(8).
Claim 1: h(ũ) = H(Φ, f Φ). By (8), (10) and the fact that τθ is the gradient of log Z
at θ [12, Thm. 2.2.1], ũn = (Qn , An ) with Qn = qθn νn and


d
X
∂ log Z
∂qθn
ai n ιi −
qθn νn = na · (ι − τθ )Qn ,
νn =
ai
An =
∂θ
∂θ
i
i
i=1
i=1
d
X

(19)

where ιi (y) = yi and ι(y) = y for any y ∈ Rd .
√ −1/2
Let L be the affine transformation on Rd given by L(y) = nΣθ (y − τθ ), and
−1/2
note that Σθ
exists because Σθ is positive-definite. By (17) and (18), this choice of L
ensures that L∗ Qn is standardised, i.e., that L∗ Qn has mean 0 and variance-covariance

9

matrix equal to the d × d identity matrix. Note that L depends on n, so we could instead
write this as Ln , but for notational simplicity we will drop the subscript. Then by (15)
and (19),
√
(20)
L∗ An = na · (ι ◦ L−1 − τθ )L∗ Qn = nf L∗ Qn ,
where f is as in the statement of the claim.
So recalling the notation L∗∗ ũn = L∗∗ (Qn , An ) = (L∗ Qn , L∗ An ), we have
h(ũ) = n−1/2 hn (ũn ) by (9) and (11)
= hn (n−1/2 ũn ) by the bilinearity of gn
= H(n−1/2 ũn ) by (12)
= H(n−1/2 L∗∗ ũn ) by (14)
= H(L∗ Qn , f L∗ Qn ) by (1) and (20).

(21)

By (16), the central limit theorem (e.g. see [13, Cor. 8.1.10]) and the fact that L∗ Qn
is standardised, L∗ Qn ⇒ Φ. Therefore,
h(ũ) = H(L∗ Qn , f L∗ Qn ) for all n, by (21)
= H(Φ, f Φ) by (13),

(22)

so the claim is proved.
Now, let v = (φ, b) ∈ T Θ be any tangent vector to Θ, not necessarily with the same
base-point as u, and let ṽ ∈ T M be the corresponding tangent vector to M.
Claim 2: aT Σθ a = bT Σφ b implies h(ũ) = h(ṽ). To prove this, assume that aT Σθ a =
1/2
1/2
T
b Σφ b, i.e. that Σθ a and Σφ b have the same Euclidean norm. Then there exists a d × d
orthogonal matrix M so that
1/2

1/2

M Σθ a = Σφ b.

(23)

Also, M∗ Φ = Φ because M is orthogonal, so
M∗∗ (Φ, f Φ) = (M∗ Φ, M∗ (f Φ)) = (M∗ Φ, (f ◦ M −1 )M∗ Φ) = (Φ, eΦ)

(24)

by (15), where e : Rd → R is given by
1/2

1/2

1/2

e(y) = f (M −1 (y)) = (Σθ a) · M −1 y = (Σθ a)T M −1 y = (Σφ b) · y

(25)

for any y ∈ Rd , by (23) and M −1 = M T (since M is orthogonal). So
h(ṽ) = H(Φ, eΦ) by Claim 1 applied to v and by (25)
= H(M∗∗ (Φ, f Φ) by (24)
= H(Φ, f Φ) by (14)
= h(ũ) by Claim 1,
which proves Claim 2.
Claim 3: There is some c > 0 so that h(ṽ) = c hF (ṽ) for all tangent vectors ṽ ∈ T M.
It is well-known [12, Thms. 2.2.1 and 2.2.5] that the Fisher information metric on the
natural parameter space is the variance-covariance matrix of the corresponding sufficient
statistic, so g F (ũ, ũ) = aT Σθ a. Alternatively, this follows easily from setting n = 1 in (19)
and combining this with (3) and the invariance of g F under sufficient statistics [1, Thm.
2.1], since these give
Z

g F (ũ, ũ) = g1F (ũ1 , ũ1 ) = aT
(y − τθ )(y − τθ )T dQ1 (y) a = aT Σθ a,
(26)

10

where ũ1 ∈ T N1 is the tangent vector to N1 corresponding to u ∈ T Θ. So Claim 2 is
equivalent to
hF (ũ) = hF (ṽ) implies h(ũ) = h(ṽ),

(27)

for all tangent vectors ũ, ṽ ∈ T M, even if they have different base-points.
Now, fix ũ to be some non-zero vector with hF (ũ) = 1, and let c = h(ũ). Note that c > 0
because g is an inner product on each tangent space so the norm of any non-zero tangent
vector is strictly positive. Then for any non-zero ṽ, hF (ṽ/hF (ṽ)) = hF (ṽ)/hF (ṽ) = 1 by
the bilinearity of g F . So hF (ũ) = hF (ṽ/hF (ṽ)) and hence, by (27), h(ũ) = h(ṽ/hF (ṽ)).
Therefore c = h(ũ) = h(ṽ/hF (ṽ)) = h(ṽ)/hF (ṽ) by the bilinearity of g, so rearranging this
equation proves the claim for all non-zero tangent vectors ṽ ∈ T M. But the claim holds
trivially for any zero tangent vector ṽ, since 0 = h(ṽ) = hF (ṽ) by the bilinearity of g and
g F , so the claim is proved.
The theorem now follows from Claim 3 and by (9), (11) and the analogous equations
for the Fisher information metrics g F , g nF and gnF , which hold by [1, eq. 4.2 and Thm.
2.1].

6

Extensions to higher-order symmetric tensors

The proof of Theorem 1 extends with almost no changes to characterise symmetric, orderk tensors ĝ and ĝn on M and Nn , respectively, that satisfy conditions closely analogous
to assumptions (A1)–(A3) of Section 4. Given such tensors ĝn , define ĥn (ũn ) =
p
k
ĝn (ũn , . . . , ũn ), where there are k copies of ũn in the right-hand side of this equation.
Assume that
ĝn (ũn , . . . , ũn ) = nk/2 ĝ1 (ũ1 , . . . , ũ1 ),

(28)

which is a generalisation of (9) from k = 2 to general k. Then as in the proof of Theorem 1,
√
ĥn (ũn ) = nĥ1 (ũ1 ) and ĥn (αũn ) = αĥn (ũn ) for any α ≥ 0 (by (28) and the multi-linearity
of ĝn ). So with ĥ in place of h, the proof of Theorem 1 implies that ĥ(ũ) = c hF (ũ) for
some c ∈ R, where hF is the norm of the Fisher information metric. Raising this equation
to the power of k gives

k/2
ĝ(ũ, . . . , ũ) = ck g F (ũ, ũ)
.

(29)

If k is odd then the left-hand side is an odd function of ũ (i.e. it changes sign when ũ is
replaced by −ũ) while the right-hand side is an even function, which is a contradiction
unless both sides vanish, so c = 0. If k is even, then since ĝ is determined by (29) (by the
polarisation formula for symmetric tensors), ĝ must be a constant times the symmetric
part of (g F )k/2 . For example, when k = 4 then there is some c′ ∈ R so that


ĝ(ũ, ṽ, w̃, m̃) = c′ g F (ũ, ṽ)g F (w̃, m̃) + g F (ũ, w̃)g F (ṽ, m̃) + g F (ũ, m̃)g F (ṽ, w̃)

for any ũ, ṽ, w̃, m̃ ∈ T M.

Remark 7. It might also be possible to adapt the proof of Theorem 1 to characterise the
higher-order Amari-Chentsov tensors, which are symmetric, order-k tensors that coincide
with the Fisher information metric when k = 2 and in general are given by an equation
similar to (3), e.g. see [2, eq. 2.4] for the k = 3 case. Claim 1 in the proof of Theorem
1 does not seem to hold for these tensors in general. However, if we replace the k/2 in
(28) by other powers and strengthen the weak continuity condition on H then it might be
possible to replace Claim 1 by ĥ(ũ) = H(KΦ, f KΦ), where K is an Edgeworth polynomial
(see [4] or [12, §4.5]). Then a symmetry argument, similar to the one in the proof of
Theorem 1, should give the desired characterisation.

11

7

Discussion

Our version of Chentsov’s theorem characterises the Fisher information metric as the
unique Riemannian metric (up to rescaling) on an exponential family M which is invariant under IID extensions and canonical sufficient statistics. We proved this by considering
metrics g on M, g n on the n-fold IID extension Mn of M, and gn on the natural exponential family Nn corresponding to Mn . Then, under the above invariance conditions, g
can be calculated in terms of gn , for any n. But for large n, the central limit theorem and
a property (16) of exponential families imply that Nn consists of distributions which are
all approximately normally distributed, so each distribution in Nn is determined to a good
approximation by its mean and variance-covariance matrix. Further, each tangent vector
to Nn is essentially a linear function f times a distribution in Nn . Combining these facts
shows that (the norm corresponding to) g is approximately equal to a simple function of
f and the mean and variance-covariance matrix of the relevant distribution in Nn . Our
regularity condition implies that this approximation becomes exact in the limit as n → ∞.
Then our main result follows from an identity (26) relating the variance-covariance matrix
to the Fisher information metric on an exponential family.
In general, Chentsov’s theorem characterizes the Fisher information metrics of statistical models as the only Riemannian metrics (up to rescaling) that are invariant under
certain, statistically important transformations. Previous studies have taken these transformations to be either all sufficient statistics or a large, regular subset of these. By
contrast, we take these statistically important transformations to be the IID extensions
and canonical sufficient statistics. This class of transformations is arguably more natural
than the class of all sufficient statistics, it is more appropriate for exponential families
and it is a relatively small class so our invariance assumptions are weaker than those of
previous studies. Our regularity assumptions also appear to be weaker than previous studies, ultimately due to the fact that our approach only requires us to study a collection of
finite-dimensional models, rather than an infinite-dimensional model.
We have given a new characterisation of the Fisher information metric on an exponential family and we have shown that this result is an intuitive consequence of the central limit
theorem. The main limitation of this paper is that our main result is only proved for exponential families. However, exponential families are an important class of statistical models,
being well studied and widely used in applications. Also, our proof treats discrete and continuous models in a uniform way, so there is some hope that our approach can be adapted
to give a proof of Chentsov’s theorem for general statistical models. Lastly, our focus on
exponential families complements the focus of Bauer et al. [7] on diffeomorphism-invariant
metrics, since (curved) exponential families are essentially the only statistical models which
have smooth sufficient statistics that are not diffeomorphisms, by the Pitman–Koopman–
Darmois theorem [5].

A The invariant and parameterisation-dependent definitions of the Fisher information metric coincide
This section proves (in the notation of Section 2) that the invariant definition (3) of the
Fisher information metric reduces to the usual parameterisation-dependent definition given
by (31), below.
Given any tangent vectors u = (θ, a) and v = (θ, b) in Tθ Θ, let ũ = (P, A) and
ṽ = (P, B) be the corresponding tangent vectors in TP M, where P = pθ µ. Then by (2),
A=

d
X
i=1

so dA/dP =

Pd

i=1

d

ai

d

X ai ∂pθ
X ∂ log pθ
∂pθ
ai
µ=
pθ µ =
P
∂θi
p ∂θi
∂θi
i=1 θ
i=1

ai (∂/∂θi ) log pθ , and similarly for dB/dP . Substituting these into (3)

12

gives
g F (ũ, ṽ) =

d
X

i,j=1

ai b j

Z 

∂ log pθ
∂θi



∂ log pθ
∂θj



pθ dµ = aT ḡθF b,

where ḡθF is the d × d matrix with (i, j)th entry


Z 
∂ log pθ
∂ log pθ
F
[ḡθ ]ij =
pθ dµ,
∂θi
∂θj

(30)

(31)

for any i, j = 1, . . . , d. Therefore the invariant definition (3) reduces to the usual,
parameterisation-dependent definition (31) for the Fisher information metric [1, eq. 2.6].
Remark 8. The metric ḡ F on Θ is just the pull-back of the metric g F on M via the
parameterisation map Θ → M.

References
[1] S. Amari and H. Nagaoka. Methods of Information Geometry, volume 191 of Translations of mathematical monographs. American Mathematical Society, Providence,
2000.
[2] N. Ay, J. Jost, H. Vân Lê, and L. Schwachhöfer. Information geometry and sufficient
statistics. Probab. Theory Relat. Fields, 162:327–364, 2015.
[3] O. Barndorff-Nielsen. Information and exponential families. John Wiley & Sons, New
York, 1978.
[4] O. Barndorff-Nielsen and D. R. Cox. Edgeworth and saddle-point approximations
with statistical applications. Journal of the Royal Statistical Society Series B (Methodological), 41(3):279–312, 1979.
[5] O. Barndorff-Nielsen and K. Pedersen. Sufficient data reduction and exponential
families. Math. Scand., 22:197–202, 1968.
[6] A. R. Barron, J. Rissanen, and B. Yu. The minimum description length principle in
coding and modeling. IEEE Transactions on Information Theory, 44(6):2743–2760,
October 1998.
[7] M. Bauer, M. Bruveris, and P. W. Michor. Uniqueness of the Fisher-Rao metric on
the space of smooth densities. Bull. London Math. Soc., 48:499–506, 2016.
[8] V. I. Bogachev. Measure Theory, Volume I. Springer, Berlin, 2007.
[9] L. L. Campbell. An extended Cencov characterization of the information metric.
Proc. Am. Math. Soc., 98:135–141, 1986.
[10] N. N. Chentsov. Algebraic foundation of mathematical statistics. Math. Operationsforsch. statist., 9:267–276, 1978.
[11] F. G. Friedlander and M. Joshi. Introduction to the Theory of Distributions (second
edition). Cambridge University Press, Cambridge UK, 1998.
[12] R. E. Kass and P. W. Vos. Geometrical Foundations of Asymptotic Inference. John
Wiley & Sons, New York, 1997.
[13] M. M. Meerschaert and H.-P. Scheffler. Limit Distributions for Sums of Independent
Random Vectors: Heavy Tails in Theory and Practice. Wiley series in probability and
statistics. John Wiley & Sons, New York, 2001.

13

[14] G. Pistone and C. Sempri. An infinite-dimensional geometric structure on the space
of all the probability measures equivalent to a given one. Ann. Stat., 23(5):1543–1561,
1995.
[15] J. Rissanen. Fisher information and stochastic complexity. IEEE Transactions on
Information Theory, 42(1):40–47, January 1996.
[16] N. N. Čencov. Statistical Decision Rules and Optimal Inference, volume 53 of Translations of Mathematical Monographs. American Mathematical Society, Providence,
1982.

14

