Information-geometrical characterization of
statistical models which are
statistically equivalent to probability simplexes

arXiv:1701.07736v2 [cs.IT] 8 Feb 2017

Hiroshi Nagaoka
Graduate School of Informatics and Engineering
The University of Electro-Communications
Chofu, Tokyo 182-8585, Japan
Email: nagaoka@is.uec.ac.jp

Abstract—The probability simplex is the set of all probability distributions on a finite set and is the most fundamental
object in the finite probability theory. In this paper we give
a characterization of statistical models on finite sets which are
statistically equivalent to probability simplexes in terms of αfamilies including exponential families and mixture families. The
subject has a close relation to some fundamental aspects of
information geometry such as α-connections and autoparallelity.

II. S TATEMENT OF

THE MAIN RESULT

We begin with giving some basic definitions which are
necessary to state our problem.
For an arbitrary finite set X , let P(X ) and P(X ) be
the sets of probability distributions and of strictly positive
probability distributions on X ;
X
P(X ) := {p | p : X → [0, 1],
p(x) = 1}
x

I. A N

INTRODUCTORY EXAMPLE

Let X = {0, 1, 2} and let M = {pλ | 0 < λ < 1} be the
set of probability distributions on X of the form
pλ = (pλ (0), pλ (1), pλ (2)) = (λ, (1 − λ)/2, (1 − λ)/2) .
The statistical model M has the following three properties.
Firstly, it is a mixture family since
pλ = λ (1, 0, 0) + (1 − λ) (0, 1/2, 1/2).
Secondly, it is an exponential family since
log pλ = θF − ψ(θ),
where θ = log(2λ/(1 − λ)), (F (0), F (1), F (2)) = (1, 0, 0)
and ψ(θ) = − log(1 − λ)/2 = log(2 + eθ ). Lastly, M is
statistically equivalent to the 1-dimensional open probability
simplex P1 = {(λ, 1 − λ) | 0 < λ < 1} in the sense that there
exist a channel V from {0, 1} to X and a channel W from
X to {0, 1} such that M is the set of output distributions of
V for input distributions in P1 and that V is invertible by W .
The matrix representations of these channels are given by




1 0
1 0 0
V = 0 1/2 , W =
.
0 1 1
0 1/2

Note that the invertibility W V = I holds.
Our aim is to show the equivalence between the first two
properties and the last one.

P(X ) := {p | p : X → (0, 1),

X

p(x) = 1}.

x

In particular, let for an arbitrary positive integer d
P d := P({0, 1, . . . , d})
Pd := P({0, 1, . . . , d}),
which we call the d-dimensional (closed and open) probability
simplexes.
A mapping Γ : P(X ) → P(Y ), where X and Y are
finite sets, is called a Markov map when there exists a channel
W (y|x) from X to Y such that, for any p ∈ P(X ),
X
Γ(p) =
W ( · |x)p(x).
x

i.e., Γ(p) is the output distribution of the channel W corresponding to the input distribution p. Note that a Markov
map is affine; Γ(λp + (1 − λ)q) = λΓ(p) + (1 − λ)Γ(q) for
∀p, q ∈ P(X ) and 0 ≤ ∀λ ≤ 1.
Let M and N be smooth submanifolds (statistical models)
of P(X ) and P(Y ), respectively. When there exist a pair
of Markov maps Γ : P(X ) → P(Y ) and ∆ : P(Y ) →
P(X ) such that their restrictions Γ|M and ∆|N are bijections
between M and N and are the inverse mappings of each other,
we say that M and N are Markov equivalent or statistically
equivalent and wite as M ≃ N .
The aim of this paper is to give a characterization of statistical models which are statistically equivalent to probability
simplexes. The main result is as follows.
Theorem 1 For an arbitrary smooth submanifold M of
P(X ), the following conditions are mutually equivalent.

For an arbitrary α ∈ R, define a function L(α) : R+ (=
(0, ∞)) → R by1
 1−α
(α 6= 1)
u 2
(α)
(5)
L (u) =
log u (α = 1).

(i) M ≃ Pd , where d = dim M .
(ii) M is an exponential family and is a mixture
family.
(iii) ∃α 6= ∃β, M is an α-family and is an β-family.
(iv) ∀α, M is an α-family.
Explanation of exponential family, mixture family and αfamily for arbitrary α ∈ R as well as the proof of the theorem
will be presented in subsequent sections. Here we only give
a few remarks on condition (i). Firstly, (i) is equivalent to the
condition that ∃d′ , M ≃ Pd′ , since if M ≃ Pd′ then M and
Pd′ must be diffeomorphic, so that dim M = dim Pd′ = d′ .
Secondly, (i) is equivalent to the condition M ≃ P d , where
M denotes the topological closure of M , and means that M
is the set of output distributions of an invertible (erro-free)
channel.
III. S OME

FACTS ABOUT CONDITION

(i)

From the definition of the relation ≃, condition (i) implies
that there exist Γ : P(X ) → P d and ∆ : P d →
P(X ) satisfying Γ ◦ ∆ = id (the identity on P d ). Let
{q0 , q1 , . . . , qd } ⊂ P(X ) be defined by
∆(δi ) = qi , ∀i ∈ {0, 1, . . . , d},

(1)

where δi is the delta distributions on {0, 1, . . . , d} concentrated
on i. Then it is easy to see, as is shown in Lemma 9.5 and its
“Supplement” of [1] where our ∆ is called a congruent embedding (of P d into P(X )), that the supports Ai := supp (qi )
constitute a partition of X in the sense that
Ai ∩ Aj = φ if i 6= j, and

d
[

Ai = X ,

(2)

i=0

The function L(α) is naturally extended to a mapping
(R+ )X → RX (f 7→ L(α) (f )) by


L(α) (f ) (x) = L(α) (f (x)).
(6)
For a submanifold M of P(X ), its denormalization M̃ is
defined by

M̃ := τ p | p ∈ M and τ ∈ R+ ,
(7)

where τ p denotes the function X ∋ x 7→ τ p(x) ∈ R+ . The
denormalization is an extended
Pmanifold obtained by relaxing
the normalization constraint x p(x) = 1. Obviously, M̃ is
f ), and P(X
f ) = (R+ )X is an open
a submanifold of P(X
X
subset of R . When the image
o
n
L(α) (M̃ ) = L(α) (τ p) p ∈ M and τ ∈ R+

forms an open subset of an affine subspace, say Z, of RX , M
is called an α-family. In this paper, it is assumed for simplicity
that M is maximal in the sense that

L(α) (M̃ ) = Z ∩ L(α) (R+ )X .
(8)
Since it follows from the definition (5) of L(α) that


(R+ )X (α 6= 1)
L(α) (R+ )X =
RX
(α = 1),

(8) is written as

and the left inverse Γ of ∆ is represented as
Γ(p) =

d
X

L
p(Ai ) δi , ∀p ∈ P(X ),

where p(Ai ) := x∈Ai p(x). In addition, condition (i) implies
M = ∆(Pd ) := {∆(λ) | λ ∈ Pd }, so that from (1) we have
)
( d
X
(4)
λi qi (λ0 , . . . , λd ) ∈ Pd .
M=
i=0

Conversely, if a statistical model M ⊂ P(X ) is represented
in the form (4) by a collection of d + 1 distributions {qi } on
X whose supports {Ai } constitute a partition of X , then we
see that M satisfies condition (i) by defining ∆ and Γ by (1)
and (3). Thus a necessary and sufficient condition for (i) is
obtained, which will be used in later arguments to prove the
theorem.
IV. α- FAMILY, e- FAMILY

AND

(M̃ ) =

(3)

i=0

P

(α)

m- FAMILY

Following the way developed in [5] (see also [3], [4]),
we give the definition of α-family, which includes that of
exponential family and mixture family as special cases.



Z ∩ (R+ )X
Z

(α 6= 1)
(α = 1).

(9)

Note that, as is pointed out in section 2.6 of [4], an affine
subspace Z satisfying (9) must be a linear subspace when
α 6= 1. Note also that P(X ) is an α-family for ∀α ∈ R,
corresponding to the case when Z = RX .
When α = 1, the notion of α-family is equivalent to that
of exponential family, whose general form is M = {pθ | θ =
(θ1 , . . . , θd ) ∈ Rd } such that
#
"
d
X
i
θ Fi (x) − ψ(θ) ,
(10)
pθ (x) = exp C(x) +
i=1

where C, F1 , . . . , Fd are functions on X and ψ is a function
on Rd defined by
#
"
d
X
X
i
θ Fi (x) .
(11)
ψ(θ) = log
exp C(x) +
x

i=1

1 L(α) (u) can be replaced with aL(α) (u) + b by arbitrary constants a 6= 0
and b, possibly depending on α. In [3], [4], [5], these constants are properly
chosen so that the ±α-duality and the limit of α → 1 can be treated in a
convenient way.

When α = −1, on the other hand, the notion of α-family
is equivalent to that of mixture family, whose general form is
M = {pθ | θ = (θ1 , . . . , θd ) ∈ Θ} such that
pθ (x) = C(x) +

d
X

i

θ Fi (x),

(12)

i=1

pθ (x) =

j=0

2
o 1−α
.
ξ (θ)Fj (x)

j

(13)

See §2.6 of [4] for further details.
V. P ROOF

OF

Let α be an arbitrary real number such that α 6= 1. Since
L(α) (0) = 0 in this case, it follows from the disjointness of
the supports of {qi } that
!
X 1−α
X
(α)
λi 2 L(α) (qi )
λi qi =
L
i

for any (λ0 , . . . , λd ) ∈ (R+ )d+1 . From this we have
L(α) (M̃ )
)
( d
X 1−α
(α)
+ d+1
2
λi L (qi ) (λ0 , . . . , λd ) ∈ (R )
=
i=0

=

ξi L(α) (qi ) (ξ0 , . . . , ξd ) ∈ (R+ )d+1

i=0

)

= Z ∩ (R+ )X ,
where Z is the (d + 1)-dimensional linear subspace of RX
spanned by L(α) (qi ), i ∈ {0, 1, . . . , d}. This proves that M is
an α-family for any α 6= 1.
Let α = 1. For any x ∈ X , we have
!
!
X
X
(1)
λi qi (x)
λi qi (x) = log
L
i

i

= log(λj qj (x))
= log λj + log qj (x)
X
(log λi + log qi (x)) 1Ai (x),
=
i

(

C+

i=0
d
X

+ d+1

(log λi )1Ai (λ0 , . . . , λd ) ∈ (R )
ξi 1Ai (ξ0 , . . . , ξd ) ∈ R

d+1

i=0

)

)

,

which is an affine subspace of RX . This proves that M is a
1-family (an exponential family).
The implication (i) ⇒ (iv) has thus been proved.
OF

(ii), (iii) AND (iv)

The implications (iv) ⇒ (ii) ⇒ (iii) are obvious. To see (iii)
⇒ (iv), some results of information geometry are invoked.

(i) ⇒ (iv)

i=0

( d
X

=

C+

d
X

VI. E QUIVALENCE

Assume (i), which implies that there exists a collection of
d + 1 probability distributions {qi } ⊂ P(X ) whose supports
{Ai } constitute a partition of X and that M is represented
as (4). Then the denormalization M̃ is represented as
)
( d
X
(14)
λi qi (λ0 , . . . , λd ) ∈ (R+ )d+1 .
M̃ =

i

L(1) (M̃ )
(
=

where F1 , . . . , Fd are functions on X satisfying
P
d
x Fi (x) = 0 and Θ := {θ ∈ R | ∀x, pθ (x) > 0}.
When α 6= 1, the general form of α-family M = {pθ | θ =
(θ1 , . . . , θd ) ∈ Θ} is
d
nX

where j denotes the element of {0, 1, . . . P
, d} such that x ∈ Aj .
Letting C ∈ RX be defined by C(x) = i (log qi (x))1Ai (x),
we have

Remark 1: The notion of affine connections appears only in
this section. Since the implication (ii) ⇒ (i) will be proved
in the next section without using affine connections (at least
explicitly), we do not need them in proving the equivalence
of the conditions of Theorem 1 except for (iii).
We first introduce some concepts from general differential
geometry. Let S be a smooth manifold and denote by T (S)
the set of smooth vector fields on S. Here, by a vector field
on S we mean a mapping, say X, such that X : S ∋ p 7→
Xp ∈ Tp (S), where Tp (S) denotes the tangent space of S
at p. An affine connection on S is represented by a mapping
∇ : T (S)×T (S) ∋ (X, Y ) 7→ ∇X Y ∈ T (S), which is called
a covariant derivative, satisfying certain conditions. Let M be
a smooth submanifold of S. Then ∇ is naturally defined on
T (M ) × T (M ), so that ∇X Y is defined for any vector fields
on M . However, the value ∇X Y in this case is a mapping
M ∋ p 7→ (∇X Y )p ∈ Tp (S) in general and is not a vector
field on M (i.e., ∇X Y 6∈ T (M )) unless
(∇X Y )p ∈ Tp (M ), ∀p ∈ M.

(15)

When (15) holds for ∀X, Y ∈ T (M ), M is said to be
autoparallel w.r.t. ∇ or ∇-autoparallel in S.
Let ∇, ∇′ and ∇′′ be affine connection on S for which there
exists a real number a satisfying2
∇′′ = a∇ + (1 − a)∇′ .

(16)

If a submanifold M is ∇-autoparallel and ∇′ -autoparallel,
then it is also ∇′′ -autoparallel. This implication is obvious
from (∇′′X Y )p = a(∇X Y )p + (1 − a)(∇′X Y )p and the
autoparallelity condition (15), which will be invoked later.
As was independently introduced by Čencov [1] and Amari
[2], a one-parameter family of affince connections, which are
called the α-connections (α ∈ R), are defined on a manifold
2 For arbitrary affine connections ∇ and ∇′ , their affine combination a∇ +
(1 − a)∇′ always becomes an affine connection.

of probability distributions. After Amari’s notation, the αconnection is written in the form of affine combination
1 + α (1) 1 − α (−1)
∇ +
∇
,
(17)
∇(α) =
2
2
which implies that
∇(γ) =

γ − β (α) α − γ (β)
∇ +
∇
α−β
α−β

(18)

for any α, β, γ ∈ R such that α 6= β.
When a submanifold M of S is autoparallel w.r.t. the αconnection in S, we say that M is α-autoparallel in S.
Since (18) is of the form (16), it follows that if M is αautoparallel and β-autoparallel in S for some α 6= β, then it
is γ-autoparallel in S for all γ ∈ R. On the other hand, it
was shown in [5] (see also section 2.6 of [4]) that, for any
submanifold M of P(X ) and for any real number α, M
is an α-family if and only if M is α-autoparallel in P(X ).
Combination of these two results proves (iii) ⇒ (iv).
Remark 2: Since the e-connection and the m-connection are
dual w.r.t. the Fisher information metric [3], [4], [5], condition
(ii) is a special case of doubly autoparallelity introduced by
Ohara; see [6], [7] and the reference cited there. It is pointed
out in [7] that the α-autoparallelity for all α follows from that
for α = ±1.
VII. P ROOF

OF

(ii) ⇒ (i)

Assume (ii), which means that there exist two affine subspaces Z (e) and Z (m) of RX such that
L(e) (M̃ ) = {log µ | µ ∈ M̃ } = Z (e)

(19)

L(m) (M̃ ) = M̃ = Z (m) ∩ (R+ )X ,

(20)

(e)

(1)

(m)

(−1)

(e)

where L := L and L
:= L
. Let V
and V (m)
(e)
be the linear spaces of translation vectors of Z and Z (m) ,
respectively, so that we have Z (e) = f +V (e) for any f ∈ Z (e)
and Z (m) = g + V (m) for any g ∈ Z (m)3 .
Lemma 1 V (e) is closed w.r.t. multiplication of functions;
i.e., a, b ∈ V (e) ⇒ ab ∈ V (e) , where the product ab
is defined by (ab)(x) = a(x)b(x).
Proof. The map
Φ := L(e) |M̃ : M̃ ∋ µ 7→ log µ ∈ Z (e)
(m)

(e)
(dΨa,ν )µ ◦ (dΦ)−1
∋ b 7→ ab ∈ V (e) .
ν :V

This proves that a, b ∈ V (e) ⇒ ab ∈ V (e) .
Lemma 2 V (e) contains the constant functions on X .
Proof. From the definition (7) of M̃ , for any µ ∈ M̃ and
any positive constant τ = ec , we have τ µ ∈ M̃ . This implies
that both log µ and log(τ µ) belong to Z (e) , and hence the
translation log(τ µ) − log µ = log τ = c belongs to V (e) .
These two lemmas state that V (e) is a subalgebra of the
commutative algebra RX with the unit element 1 (: the
constant function x 7→ 1) of RX contained in V (e) . From
a well known result on such subalgebras4 , it is concluded
that there exists a partition {Ai }di=0 of X such that
)
( d
X
d+1
(e)
.
(21)
ci 1Ai (c0 , . . . , cd ) ∈ R
V
=
i=0

Let an element p0 of M (⊂ M̃ ) be arbitrarily fixed. Then we
have
Z (e) = log p0 + V (e) .
(22)
From (19), (21) and (22) and the disjointness of {Ai }, we
have
M̃ = {µ | log µ ∈ Z (e) }
= {µ | log µ − log p0 ∈ V (e) }
n
= µ ∃(c0 , . . . , cd ) ∈ Rd+1 ,
log µ = log p0 +

+ X

is a diffeomorphism from M̃ = Z
∩ (R ) , which is an
open subset of Z (m) , onto Z (e) . The differential map of Φ at
a point µ ∈ M̃ is defined by
 dµ(t)

d
(dΦ)µ
= Φ(µ(t))
dt t=0
dt
t=0
for any smooth curve µ(t) in M̃ and is represented as
(dΦ)µ : V (m) ∋ f 7→

This gives a linear isomorphism from V (m) onto V (e) . Therefore, for any two points µ, ν ∈ M̃ , we can define
µa
(e)
∈ V (e) .
(dΦ)ν ◦ (dΦ)−1
∋ a 7→
µ :V
ν
This means that, for any a ∈ V (e) and any µ, ν ∈ M̃ , we have
µa
(e)
. For arbitrary a ∈ V (e) and ν ∈ M̃ , let us define a
ν ∈ V
map Ψa,ν by
µa
Ψa,ν : M̃ ∋ µ 7→
∈ V (e) .
ν
Then its differential at a point µ ∈ M̃ is given by
ga
(dΨa,ν )µ : V (m) ∋ g 7→
∈ V (e) .
ν
Composing this map with the inverse of
g
(dΦ)ν : V (m) ∋ g 7→ ∈ V (e) ,
ν
we have

f
∈ V (e) .
µ

3 Actually, Z (m) is a linear space as mentioned in section IV, and therefore
Z (m) = V (m) .

=
=

(

(

d
X
i=0

p0

d
X

o
c i 1 Ai , ,

ci

e 1Ai (c0 , . . . , cd ) ∈ R

d+1

i=0

d
X
i=0

+ d+1

λi qi (λ0 , . . . , λd ) ∈ (R )

)

)

,

4 Although various mathematical extensions of this result including infinitedimensional and/or noncommutative versions are known, the author of the
present paper could find no appropriate reference describing the result for the
finite-dimensional commutative case with an elementary proof. So, we give a
proof in the appendix for the readers’ sake.

where

1
p 0 1 Ai ,
qi :=
p0 (Ai )

Proof. Let
i ∈ {0, . . . , d}.

Then {qi } are probability distributions on X whose supports
are supp (qi ) = Ai , and
M = M̃ ∩ P(X )
)
( d
X
λi qi (λ0 , . . . , λd ) ∈ Pd .
=
i=0

Since this is the same form as (4), condition (i) has been
derived.
VIII. C ONCLUSION
We have shown Theorem 1 which gives an informationgeometrical characterization of statistical models on finite
sample spaces which are statistically equivalent to open probability simplexes Pd . The statistical equivalence (also called
the Markov equivalence) to probability simplexes played a
crucial role in Čencov’s pioneering work [1] on information
geometry, where the notions of Fisher information metric
and the α-connections were characterized in terms of the
statistical equivalence. The present work shed another light on
the relation between the statistical equivalence and information
geometry.
ACKNOWLEDGMENT
This work was partially supported by JSPS KAKENHI
Grant Number 16K00012.


B := f −1 (λ) | λ ∈ R, f ∈ V

⊂ 2X ,

(24)

which is the totality of the level sets f −1 (λ) = {x | f (x) =
λ} ⊂ X of functions in V . We first show that, for any B ⊂
X,
B ∈ B ⇔ 1B ∈ V.
(25)
Since ⇐ is obvious, it suffices to show ⇒. Assume B ∈ B,
so that B = f −1 (λ) for some f ∈ V and λ ∈ R. When B
is the empty set φ, we have 1B = 0 ∈ V . So we assume
B 6= φ, which means that λ ∈ f (X ). Let the elements of
f (X ) be λ0 , λ1 , . . . λk , where λ0 = λ and λi 6= λj if i 6= j,
Pk
and let Bi := f −1 (λi ). Then we have f = i=0 λj 1Bi with
B0 = B. Let a(t) = a0 tk + a1 tk−1 + · · · + ak be a polynomial
satisfying a(λ0 ) = 1 and a(λi ) = 0 for any i 6= 0. Explicitly,
a(t) is expressed as
a(t) =
It follows that
a(f ) =

k
X

k
Y
t − λi
.
λ − λi
i=1 0

a(λi )1Bi = 1B0 = 1B .

i=0

In addition, a(f ) = a0 f k + a1 f k−1 + · · · + ak belongs to V
since V is a subalgebra of RX with 1 ∈ V . Hence we have
1B ∈ V .
Using (25), we see that

R EFERENCES

X ∈ B,

(26)

B ∈ B ⇒ B c ∈ B,
B1 , B2 ∈ B ⇒ B1 ∩ B2 ∈ B

(27)
(28)

[1] N. N. Čencov (Chentsov), Statistical Decision Rules and Optimal Inference, AMS, 1982 (original Russian edition: Nauka, Moscow, 1972).
[2] S. Amari, “Differential geometry of curved exponential families—
as
curvature and information loss”, The Annals of Statistics, 10, 357–385,
1982.
1X = 1 ∈ V ⇒X ∈ B,
(29)
[3] S. Amari, Differential-Geometrical Methods in Statistics, Springer, LecB ∈ B ⇒ 1B ∈ V ⇒ 1B c =1 − 1B ∈ V
ture Notes in Statistics 28, 1985.
[4] S. Amari and H. Nagaoka, Methods of information geometry, AMS &
⇒ B c ∈ B,
(30)
OUP, 2000.
[5] H. Nagaoka and S. Amari, “Differential geometry of smooth
B1 , B2 ∈ B ⇒ 1B1 , 1B2 ∈ V ⇒ 1B1 ∩B2 = 1B1 1B2 ∈ V
families of probability distributions”, Technical Report METR 82⇒ B1 ∩ B2 ∈ B.
(31)
7, Dept. of Math. Eng. and Instr. Phys, Univ. of Tokyo, 1982.
(http://www.keisu.t.u-tokyo.ac.jp/research/techrep/data/1982/METR82-07.pdf)
Properties (26)-(28) implies that B is an additive class of
[6] A. Ohara, “Information geometric analysis of an interior point method
for semidefinite program- ming”, Geometry in Present Day Science (eds.
sets (σ-algebra) on the finite entire set X . Therefore, B is
O. E. Barndorff-Nielsen and E. B. V. Jensen), pp.49-74, World Scientific,
generated by a partition {A1 , · · · , An } of X in the sense that
1999.
every element of B is the union of some (or no) elements
[7] A. Ohara, “Geodesics for dual connections and means on symmetric
cones”, Integr. equ. oper. theory, 50, 537–548, 2004.
of {A1 , · · · , An }. Recalling the definition (24) of B, we

A PPENDIX
Proposition Let X be a finite set and V be a subalgebra
of RX containing the constant functions. Then there exists a
partition {Ai }ni=1 of X such that
)
( n
X
n
.
(23)
ci 1Ai (c1 , . . . , cn ) ∈ R
V =
i=1

conclude (23).

