Understanding Boltzmann Machine and Deep Learning via
A Confident Information First Principle
Xiaozhao Zhao

0.25eye@gmail.com

School of Computer Science and Technology, Tianjin University
Tianjin, 300072, China

arXiv:1302.3931v7 [] 9 Oct 2013

Yuexian Hou

krete1941@gmail.com

School of Computer Science and Technology, Tianjin University
Tianjin, 300072, China

Qian Yu

yqcloud@gmail.com

School of Computer Software, Tianjin University
Tianjin, 300072, China

Dawei Song

dawei.song2010@gmail.com

School of Computer Science and Technology, Tianjin University
Tianjin, 300072, China
and Department of Computing, The Open University
Milton Keynes, MK7 6AA, UK

Wenjie Li

cswjli@comp.polyu.edu.hk

Department of Computing, The Hong Kong Polytechnic University
Hung Hom, Kowloon, Hong Kong, China

Editor:

Abstract
Typical dimensionality reduction methods focus on directly reducing the number of random variables while retaining maximal variations in the data. In this paper, we consider
the dimensionality reduction in parameter spaces of binary multivariate distributions. We
propose a general Confident-Information-First (CIF) principle to maximally preserve parameters with confident estimates and rule out unreliable or noisy parameters. Formally,
the confidence of a parameter can be assessed by its Fisher information, which establishes
a connection with the inverse variance of any unbiased estimate for the parameter via the
Cramér-Rao bound. We then revisit Boltzmann machines (BM) and theoretically show
that both single-layer BM without hidden units (SBM) and restricted BM (RBM) can be
solidly derived using the CIF principle. This can not only help us uncover and formalize
the essential parts of the target density that SBM and RBM capture, but also suggest
that the deep neural network consisting of several layers of RBM can be seen as the layerwise application of CIF. Guided by the theoretical analysis, we develop a sample-specific
CIF-based contrastive divergence (CD-CIF) algorithm for SBM and a CIF-based iterative
projection procedure (IP) for RBM. Both CD-CIF and IP are studied in a series of density
estimation experiments.
Keywords: Boltzmann Machine, Deep Learning, Information Geometry, Fisher Information, Parametric Reduction

1

1. Introduction
Recently, deep learning models (e.g., Deep Belief Networks (DBN)(Hinton and Salakhutdinov, 2006), Stacked Denoising Auto-encoder (Ranzato et al., 2006), Deep Boltzmann
Machine (DBM) (Salakhutdinov and Hinton, 2012) and etc.) have drawn increasing attention due to their impressive results in various application areas, such as computer vision
(Bengio et al., 2006; Ranzato et al., 2006; Osindero and Hinton, 2007), natural language processing (Collobert and Weston, 2008) and information retrieval (Salakhutdinov and Hinton,
2007a,b). Despite of these practical successes, there have been debates on the fundamental
principle of the design and training of those deep architectures. One important problem is
the unsupervised pre-training, which would fit the parameters to better capture the structure
in the input distribution (Erhan et al., 2010). From the probabilistic modeling perspective,
this process can be interpreted as an attempt to recover a set of model parameters for a
generative neural network that would well describe the distribution underlying the observed
high-dimensional data. In general, to sufficiently depict the original high-dimensional data
requires a high-dimensional parameter space. However, overfitting usually occur when the
model is excessively complex. On the other hand, Dauphin and Bengio (2013) empirically
shows the failure of some big neural networks in leveraging the added capacity to reduce
underfitting.
Hence it is important to uncover and understand what the first principle would be on
reducing the dimensionality of the parameter space concerning the sample size. This question is also recognized by Erhan et al. (2010). They empirically show that the unsupervised
pre-training acts as a regularization on parameters in a way that the parameters are set in a
region, from which better basins of attraction can be reached. The regularization on parameters could be seen as a kind of dimensionality reduction procedure on parameter spaces
by restricting those parameters in a desired region. However, the intrinsic mechanisms
behind the regularization process are still unclear. Thus, further theoretical justifications
are needed in order to formally analyze what the essential parts of the target density that
the neural networks can capture. An indepth investigation into this question will lead to
two significant results: 1) a formal explanation on what exactly the neural networks would
perform in the pre-training process; 2) some theoretical insights on how to do it better.
Duin and Peȩkalska (2006) empirically show that the sampling density of a given dataset
and the resulting complexity of a learning problem are closely interrelated. If the initial
sampling density is insufficient, this may result in a preferred model of a lower complexity,
so that we have a satisfactory sampling to estimate model parameters. On the other hand,
if the number of samples is originally abundant, the preferred model may become more
complex so that we could have represented the dataset in more details. Moreover, this
connection becomes more complicated if the observed samples contain noises. Now the
obstacle is how to incorporate this relationship between the dataset and the preferred model
into the learning process. In this paper, we mainly focus on analyzing the Boltzmann
machines, the main building blocks of many neural networks, from a novel information
geometry (IG)(Amari and Nagaoka, 1993) perspective.
Assuming there exists an ideal parametric model S that is general enough to represent
all system phenomena, our goal of the parametric reduction is to derive a lower-dimensional
sub-model M for a given dataset (usually insufficient or perturbed by noises) by reducing
2

the number of free parameters in S. In this paper, we propose a Confident-InformationFirst (CIF) principle to maximally preserve the parameters with highly confident estimates
and rule out the unreliable or noisy parameters with respect to the density estimation of
binary multivariate distributions. From the IG point of view, the confidence 1 of a parameter
can be assessed by its Fisher information (Amari and Nagaoka, 1993), which establishes a
connection with the inverse variance of any unbiased estimate for the considered parameter
via the Cramér-Rao bound (see Rao, 1945). It is worth emphasizing that the proposed CIF
as a principle of parametric reduction is fundamentally different from the traditional feature
reduction (or feature extraction) methods (Fodor, 2002; Lee and Verleysen, 2007). The
latter focus on directly reducing the dimensionality on feature space by retaining maximal
variations in the data, e.g., Principle Components Analysis (PCA) (Jolliffe, 2002), while CIF
offers a principled method to deal with high-dimensional data in the parameter spaces by
a strategy that is universally derived from the first principle, independent of the geometric
metric in the feature spaces.
The CIF takes an information-oriented viewpoint of statistical machine learning. The
information 2 is rooted on the variations (or fluctuations) in the imperfect observations
(due to insufficient sampling, noise and intrinsic limitations of the “observer”) and transmits throughout the whole learning process. This idea is also well recognized in modern
physics, as stated in Wheeler (1994): “All things physical are information-theoretic in origin and this is a participatory universe...Observer participancy gives rise to information;
and information gives rise to physics.”. Following this viewpoint, Frieden (2004) unifies
the derivation of physical laws in major fields of physics, from the Dirac equation to the
Maxwell-Boltzmann velocity dispersion law, using the extreme physical information principle (EPI). The information used in this unification is exactly the Fisher information (Rao,
1945), which measures the quality of any measurement(s).
In terms of statistical machine learning, the aim of this paper is of two folds: a) to
incorporate the Fisher information into the modelling of the intrinsic variations in the data
that give rise to the desired model, by using the IG framework (Amari and Nagaoka, 1993);
b) to show by examples that some existing probabilistic models, e.g., SBM and RBM,
comply with the CIF principle and can be derived from it. The main contributions are:
1. We propose a general CIF principle for parameter reduction to maximally preserve
the parameters of high confidence and eliminate the unreliable parameters of binary
multivariate distributions in the framework of IG. We also give a geometric interpretation of CIF by showing that it can maximally preserve the expected information
distance.
2. The implementation of CIF, that is, the derivation of probabilistic models, is illustrated by revisiting two widely used Boltzmann machines, i.e., Single layer BM without hidden units (SBM) and restricted BM (RBM). And the deep neural network
consisting of several layers of RBM can be seen as the layer-by-layer application of
CIF.
1. Note that, in this paper, the meaning of confidence is different from the common concept degree of
confidence in statistics.
2. There are many kinds of “information” defined for a probability distribution p(x), e.g., entropy, Fisher
information. The entropy is a global measure of smoothness in p(x). Fisher information is a local
measure that is sensitive to local rearrangement of points (x, p(x)).

3

3. Based on the above theoretical analysis, a CIF-based iterative projection procedure
(IP) inherent in the learning of RBM is uncovered. And traditional gradient-based
methods, e.g., maximum-likelihood (ML) and contrastive divergence (CD) (CarreiraPerpinan and Hinton, 2005), can be seen as approximations of IP. Experimental results
indicate that IP is more robust against sampling biases, due to its separation of the
positive sampling process and the gradient estimation.
4. Beyond the general CIF, we propose a sample-specific CIF and integrate it into the CD
algorithm for SBM to confine the parameter space to a confident parameter region
indicated by samples. It leads to a significant improvement in a series of density
estimation experiments, when the sampling is insufficient.
The rest of the paper is organized as follows: Section 2 introduces some preliminaries of
IG. Then the general CIF principle is proposed in Section 3. In Section 4, we analyze two
implementations of CIF using the BM with and without hidden units, i.e., SBM and RBM.
After that, a sample-specific CIF-based CD learning method (CD-CIF) for SBM and a CIFbased iterative projection procedure for RBM are proposed and experimentally studied in
Section 5. Finally, we draw conclusions and highlight some future research directions in
Section 6.

2. Theoretical Foundations of Information Geometry
In this section, we introduce and develop the theoretical foundations of Information Geometry (IG) (Amari and Nagaoka, 1993) for the manifold S of binary multivariate distributions
with a given number of variables n, i.e., the open simplex of all probability distributions
over binary vector x ∈ {0, 1}n . This will lay the foundation for our theoretical deviation of
the general CIF.
2.1 Notations for Manifold S
In IG, a family of probability distributions is considered as a differentiable manifold with
certain parametric coordinate systems. In the case of binary multivariate distributions,
four basic coordinate systems are often used: p-coordinates, η-coordinates, θ-coordinates
and mixed-coordinates (Amari and Nagaoka, 1993; Hou et al., 2013). Mixed-coordinates is
of vital importance for our analysis.
For the p-coordinates [p] with n binary variables, the probability distribution over 2n
states of x can be completely specified by any 2n − 1 positive numbers indicating the
probability of the corresponding exclusive states on n binary variables. For example, the
p-coordinates of n = 2 variables could be [p] = (p01 , p10 , p11 ). Note that IG requires all
probability terms to be positive.
For simplicity, we use the capital letters I, J, . . . to index the coordinate parameters
of probabilistic distribution. An index I can be regarded as a subset of {1, 2, . . . , n}.
And pI stands for the probability that all variables indicated by I equal to one and
the complemented variables are zero. For example, if I = {1, 2, 4} and n = 4, then
pI = p1101 = P rob(x1 = 1, x2 = 1, x3 = 0, x4 = 1). Note that the null set can also be
a legal index of the p-coordinates, which indicates the probability that all variables are
zero, denoted as p0...0 .
4

Another coordinate system often used in IG is η-coordinates, which is defined by:
Y
ηI = E[XI ] = P rob{ xi = 1}

(1)

i∈I

Q

where the value of XI is given by i∈I xi and the expectation is taken with respect to
the probability distribution over x. Grouping the coordinates by their orders, η-coordinate
2 , . . . , ηn
system is denoted as [η] = (ηi1 , ηij
1,2...n ), where the superscript indicates the order
2 denotes the set of all η parameters
number of the corresponding parameter. For example, ηij
with the order number 2.
The θ-coordinates (natural coordinates) is defined by:
X
θI XI − ψ
(2)
log p(x) =
I⊆{1,2,...,n},I6=N ullSet

where ψ = − log P rob{xi = 0, ∀i ∈ {1, 2, ..., n}}. The θ-coordinate is denoted as [θ] =
(θ1i , θ2ij , . . . , θn1,...,n ), where the subscript indicates the order number of the corresponding
parameter. Note that the order indices locate at different positions in [η] and [θ] following
the convention in Amari et al. (1992).
The relation between coordinate systems [η] and [θ] is bijective (Amari et al., 1992).
More formally, they are connected by the Legendre transformation:
θI =

∂φ(η)
∂ψ(θ)
, ηI =
∂ηI
∂θI

(3)

where ψ(θ) and φ(η) meet the following identity
X
ψ(θ) + φ(η) −
θI ηI = 0

(4)

The function ψ(θ) is introduced in Eq. (2):
X
X
ψ(θ) = log(
exp{
θI XI (x)})

(5)

x

I

and hence φ(η) is the negative of entropy:
X
φ(η) =
p(x; θ(η)) log p(x; θ(η))

(6)

x

Next we introduce mixed-coordinates, which is important for our derivation of CIF. In
general, the manifold S of probability distributions could be represented by the l-mixedcoordinates (Amari et al., 1992):
i,j,...,k
2
l
[ζ]l = (ηi1 , ηij
, . . . , ηi,j,...,k
, θl+1
, . . . , θn1,...,n )

(7)

where the first part consists of η-coordinates with order less or equal to l (denoted by [η l− ])
and the second part consists of θ-coordinates with order greater than l (denoted by [θl+ ]),
l ∈ {1, ..., n}.
5

2.2 Fisher Information Matrix for Parametric Coordinates
For a general coordinate system [ξ], the ith-row and jth-col element of the Fisher information matrix for [ξ] (denoted by Gξ ) is defined as the covariance of the scores of [ξi ] and [ξj ]
(Rao, 1945), i.e.,
∂ log p(x; ξ) ∂ log p(x; ξ)
gij = E[
·
]
∂ξi
∂ξj
The Fisher information measures the amount of information in the data that a statistic
carries about the unknown parameter (Kass, 1989). The Fisher information matrix is of
vital importance to our analysis because the inverse of Fisher information matrix gives an
asymptotically tight lower bound of the covariance matrix of any unbiased estimate for the
considered parameters (Rao, 1945). Moreover, the Fisher information matrix, as a distance
metric, is invariant to re-parameterization (Rao, 1945), and can be proved to be the unique
metric that is invariant to the map of random variables corresponding to a sufficient statistic
(Amari et al., 1992; Chentsov, 1982).
Another important concept related to our analysis is the orthogonality defined by Fisher
information. Two coordinate parameters ξi and ξj are called orthogonal if and only if their
Fisher information vanishes, i.e., gij = 0, meaning that their influences to the log likelihood
function are uncorrelated. A more technical meaning of orthogonality is that the maximum
likelihood estimates (MLE) of orthogonal parameters can be independently performed.
∂ 2 ψ(θ)
The Fisher information for [θ] can be rewritten as gIJ = ∂θ
I ∂θ J , and for [η] it is
2

∂ φ(η)
g IJ = ∂η
(Amari and Nagaoka, 1993). Let Gθ = (gIJ ) and Gη = (g IJ ) be the Fisher
I ∂ηJ
information matrices for [θ]
It can be shown that Gθ and Gη are mutuPand [η] respectively.
I , where δ I = 1 if I = K and 0 otherwise (Amari
ally inverse matrices, i.e., J g IJ gJK = δK
K
and Nagaoka, 1993).
In order to generally compute Gθ and Gη , we develop the following Proposition 1 and
2. Note that Proposition 1 is a generalization of Theorem 2 in Amari et al. (1992).

Proposition 1 The Fisher information between two parameters θI and θJ in [θ], is given
by
gIJ (θ) = ηI S J − ηI ηJ
(8)
Proof in Appendix A.1
Proposition 2 The Fisher information between two parameters ηI and ηJ in [η], is given
by
X
1
g IJ (η) =
(−1)|I−K|+|J−K| ·
(9)
pK
K⊆I∩J

where | · | denotes the cardinality operator.
Proof in Appendix A.2
For example, let [p] = (p001 , p010 , p011 , p100 , p101 , p110 , p111 ) be the p-coordinates of a probability distribution with three variables. Then, the Fisher information of ηI and ηJ can be
1
1
+ p010
, and if
calculated based on Equation (9): if I = {1, 2} and J = {2, 3}, g IJ = p000
1
1
1
1
IJ
I = {1, 2} and J = {1, 2, 3}, g = −( p000 + p010 + p100 + p110 ), etc.
6

3. The General CIF Principle For Parametric Reduction
As described in Section 2.1, the general manifold S of all probability distributions over
binary vector x ∈ {0, 1}n , could be exactly represented using the parametric coordinate
systems of dimensionality 2n − 1. However, given the limited samples generated from a target distribution, it is almost infeasible to determine its coordinates in such high-dimensional
parameter space with acceptable accuracy and in reasonable time. Given a target distribution q(x) on the general manifold S, we consider the problem of realizing it by a lowerdimensionality submanifold. This is defined as the problem of parametric reduction for
multivariate binary distributions.
3.1 The General CIF Principle
In this section, we will formally illuminate the general CIF for parametric reduction. Intuitively, if we can construct a coordinate system so that the confidences (measured by Fisher
information) of its parameters entail a natural hierarchy, in which high confident parameters can be significantly distinguished from low confident ones, then the general CIF can be
conveniently implemented by keeping the high confident parameters unchanging and setting
the lowly confident parameters to neutral values. However, the choice of coordinates (or
equivalently, parameters) in CIF is crucial to its usage. This strategy is infeasible in terms
of p-coordinates, η-coordinates or θ-coordinates, since it is easy to see that the hierarchies
of confidences in these coordinate systems are far from significant, as shown by an example
later in this section.
The following propositions show that mixed-coordinates meet the requirement realizing
the general CIF. Let [ζ]l be the mixed-coordinates defined in Section 2.1. Proposition 3
gives a closed form for calculating the Fisher information matrix Gζ .
Proposition 3 The Fisher information matrix of the l-mixed-coordinates [ζ]l is given by:


A 0
(10)
Gζ =
0 B
−1
−1
−1
where A = ((G−1
η )Iη ) , B = ((Gθ )Jθ ) , Gη and Gθ are the Fisher information matrices
of [θ] and [ζ]l , respectively, and Iη is the index of the parameters shared by [η] and [ζ]l ,
l }, and J is the index set of the parameters shared by [θ] and [ζ] , i.e.,
i.e., {ηi1 , ..., ηij
θ
l
i,j,...,k
{θl+1
, . . . , θn1,...,n }.
Proof in Appendix A.3.

Proposition 4 The diagonal of A are lower bounded by 1, and that of B are upper bounded
by 1.
Proof in Appendix A.4.
According to Proposition 3 and Proposition 4, the confidences of coordinate parameters
in [ζ]l entail a natural hierarchy: the first part of high confident parameters [η l− ] are separated from the second part of low confident parameters [θl+ ], which has a neutral value
(zero). Moreover, the parameters in [η l− ] is orthogonal to the ones in [θl+ ], indicating that
7

Figure 1: By projecting a point q(x) on S to a submanifold M , the l-tailored-mixedcoordinates [ζ]lt gives a desirable M that maximally preserve the expected Fisher
information distance when projecting a ε-neighborhood centered at q(x) onto M .

we could estimate these two parts independently (Hou et al., 2013). Hence we can implement the general CIF principle for parametric reduction in [ζ]l by replacing low confident
parameters with neutral value zeros and reconstructing the resulting distribution. It turns
l
, 0, . . . , 0). We call
out that the submanifold tailored by CIF becomes [ζ]lt = (ηi1 , ..., ηij...k
[ζ]lt the l-tailored-mixed-coordinates.
To grasp an intuitive picture for the general CIF strategy and its significance w.r.t
mixed-coordinates, let us consider an example with [p] = (p001 = 0.15, p010 = 0.1, p011 =
0.05, p100 = 0.2, p101 = 0.1, p110 = 0.05, p111 = 0.3). Then the confidences for coordinates
in [η], [θ] and [ζ]2 are given by the diagonal elements of the corresponding Fisher information matrices. Applying the 2-tailored CIF in mixed-coordinates, the loss ratios of Fisher
information is 0.001%, and the ratio of the Fisher information of the tailored parameter
(θ3123 ) to the remaining η parameter with the smallest Fisher information is 0.06%. On the
other hand, the above two ratios become 7.58% and 94.45% (in η-coordinates) or 12.94%
and 92.31% (in θ-coordinates), respectively.
Next, we will restate the CIF principle in terms of the geometric perspective of submanifold projection. Let M be a smooth submanifold in S. Given a point q(x) ∈ S, the
projection of q(x) to M is the point p(x) that belongs to M and is closest to q(x) in the
sense of the Kullback-Leibler divergence (K-L divergence) from the distribution q(x) to p(x)
(Amari et al., 1992):
X
q(x)
D(q(x), p(x)) =
q(x) log
(11)
p(x)
x
Alternatively, since K-L divergence is not symmetric, the projection of q(x) to M can also
be defined as the point p(x) ∈ M that minimizes the K-L divergence from M to q(x). In
the rest of this paper, the direction of the K-L divergence used in a particular projection is
explicitly specified when there is an ambiguity.
8

The CIF entails a submanifold of S via the l-tailored-mixed-coordinates [ζ]lt . However,
there exist many different submanifolds of S. Now our question is: does there exist a
general criterion to distinguish which projection is best? If such principle does exist, is CIF
the right one? The following proposition shows that the general CIF entails a geometric
interpretation illuminated in Figure 1, which would lead to an optimal submanifold M .3
Proposition 5 Given a statistical manifold S in l-mixed-coordinates [ζ]l , let the corresponding l-tailored-mixed-coordinates [ζ]lt has k free parameters. Then, among all k-dimensional
submanifolds of S, the submanifold determined by [ζ]lt can maximally preserve the expected
information distance induced by Fisher-Rao metric.
Proof in Appendix A.5.

4. Two Implementations of CIF using Boltzmann Machine
In previous section, a general CIF is uncovered in the [ζ]l coordinates for multivariate binary
distributions. Now we consider the implementations of CIF when l equals to 2 using the
Boltzmann machines (BM). More specifically, we show that two kinds of BMs, i.e., the
single layer BM without hidden units (SBM) and the restricted BM (RBM), are indeed
instances following the general CIF principle. For each case, the application of CIF can be
interpreted in two perspectives: an algebraic and geometric interpretation.
4.1 Neural Networks as Parametric Reduction Model
Many neural networks with fixed architecture, such as SBM, RBM, high-order BM (Albizuri
et al., 1995), deep belief networks (Hinton and Salakhutdinov, 2006), have been proposed to
approximately realize the underlying distributions in different application scenarios. Those
neural networks are designed to fulfill the parametric reduction for certain tasks by specifying the number of adjustable parameters, namely the number of connection weights and the
number of biases. We believe that there exists a general criterion to design the structure of
neural submanifolds for the particular application in hand, and the problem of parametric
reduction is equivalent to the choice of submanifolds. Next, we will briefly introduce the
general BM and the gradient-based learning algorithm.
4.1.1 Introduction To The Boltzmann Machines
In general, a BM (Ackley et al., 1985) is defined as a stochastic neural network consisting of
visible units x ∈ {0, 1}nx and hidden units h ∈ {0, 1}nh , where each unit fires stochastically
depending on the weighted sum of its inputs. The energy function is defined as follows:
1
1
EBM (x, h; ξ) = − xT U x − hT V h − xT W h − bT x − dT h
2
2

(12)

3. Note that the CIF is related to but fundamentally different from the m-projection in Amari et al. (1992).
Amari et al. (1992) focuses on the problem of projecting a point Q on S to the submanifold of BM and
shows that m-projection is the point on BM that is closest to Q. Actually, the m-projection is a special
case of our [ζ]lt -projection when l is 2. In the present paper, we focus on the problem of developing a
general criterion that could help us find the optimal submanifold to project on.

9

where ξ = {U, V, W, b, d} are the parameters: visible-visible interactions (U ), hidden-hidden
interactions (V ), visible-hidden interactions (W ), visible self-connections (b) and hidden selfconnections (d). The diagonals of U and V are set to zero. We can express the Boltzmann
distribution over the joint space of x and h as below:
p(x, h; ξ) =

1
exp{−EBM (x, h; ξ)}
Z

(13)

where Z is a normalization factor.
Let B be the set of Boltzmann distributions realized by BM. Actually, B is a submanifold
of the general manifold Sxh over {x, h}. From Equation (13) and (12), we can see that
ξ = {U, V, W, b, d} plays the role of B’s coordinates in θ-coordinates (Equation 2) as follows:
h

θ1 : θ1xi = bxi , θ1 j = dhj (∀xi ∈ x, hj ∈ h)
x xj

θ2 : θ2 i

x hj

= Uxi ,xj , θ2 i

x ...xj hu ...hv

θ2+ : θmi

h hj

= Wxi ,hj , θ2 i

= Vhi ,hj (∀xi , xj ∈ x; hi , hj ∈ h)

= 0, m > 2, (∀xi , . . . , xj ∈ x; hu , . . . , hv ∈ h)

(14)

So the θ-coordinates for BM is given by:
h

xh

xx

hh

[θ]BM = (θ1xi , θ1 j , θ2 i j , θ2 i j , θ2 i j , 0, . . . , 0).
{z
} | {z }
| {z } |
1−order

(15)

orders>2

2−order

The SBM and RBM are special cases of the general BM. Since SBM has nh = 0 and all
the visible units are connected to each other, the parameters of SBM are ξsbm = {U, b} and
{V, W, d} are all set to zero. For RBM, it has connections only between hidden and visible
units. Thus, the parameters of RBM are ξrbm = {W, b, d} and {U, V } are set to zero.
4.1.2 Formulation on the Gradient-based Learning of BM
Given the sample x that generated from the underlying distribution, the maximum-likelihood
(ML) is commonly used gradient ascent method for training BM in order to maximize the
log-likelihood log p(x; ξ) of the parameters ξ (Carreira-Perpinan and Hinton, 2005). Based
on Equation (13), the log-likelihood is given as follows:
X
X
0 0
log p(x; ξ) = log
e−E(x,h;ξ) − log
e−E(x ,h ;ξ)
x0 ,h0

h

Differentiating the log-likelihood, the gradient vector with respect to ξ is as follows:
∂ log p(x; ξ) X
∂[−E(x, h; ξ)] X
∂[−E(x0 , h0 ; ξ)]
p(h0 |x0 ; ξ)
=
p(h|x; ξ)
−
∂ξ
∂ξ
∂ξ
0 0
h

(16)

x ,h

The ∂E(x,h;ξ)
can be easily calculated from Equation (12). Then we can obtain the stochastic
∂ξ
gradient using Gibbs sampling (Gilks et al., 1996) in two phases: sample h given x for the
first term, called the positive phase, and sample (x0 , h0 ) from the stationary distribution
p(x0 , h0 ; ξ) for the second term, called the negative phase. Now with the resulting stochastic
gradient estimation, the learning rule is to adjust ξ by:
∆ξ = ε ·

∂ log p(x; ξ)
∂E(x, h; ξ)
∂E(x0 , h0 ; ξ)
= ε · (−h
i0 + h
i∞ )
∂ξ
∂ξ
∂ξ
10

(17)

where ε is the learning rate, h·i0 denotes the average using the sample data and h·i∞ denotes
the average with respect to the stationary distribution p(x, h; ξ) after the corresponding
Gibbs sampling phases.4
In following sections, we will revisit two special BM, namely SBM and RBM, and theoretically show that both SBM and RBM can be derived using the CIF principle. This helps
us formalize what essential parts of the target density the SBM and RBM capture.
4.2 The CIF-based Derivation of Boltzmann Machine without Hidden Units
Given any underlying probability distribution q(x) on the general manifold S over {x}, the
logarithm of q(x) can be represented by a linear decomposition of θ-coordinates as shown
in Equation (2). Since it is impractical to recognize all coordinates for the target distribution, we would like to only approximate part of them and end up with a k-dimensional
submanifold M of S, where k ( 2nx − 1) is the number of free parameters. Here, we set
k to be the same dimensionality as SBM, i.e., k = nx (n2x +1) , so that all candidate submanifolds are comparable to the submanifold endowed by SBM (denoted as Msbm ). Next, the
rationale underlying the design of Msbm can be illuminated using the general CIF from two
perspectives, algebraically and geometrically.
4.2.1 SBM as 2-tailored-mixed-coordinates
2 , θ i,j,k , . . . , θ 1,...,nx ). Applying
Let the 2-mixed-coordinates of q(x) on S be [ζ]2 = (ηi1 , ηij
nx
3
the general CIF on [ζ]2 , our parametric reduction rule is to preserve the high confident
part parameters [η 2− ] and replace low confident parameters [θ2+ ] with a fixed neutral value
2 , 0, . . . , 0), as the
zero. Thus we derive the 2-tailored-mixed-coordinates: [ζ]2t = (ηi1 , ηij
optimal approximation of q(x) by the k-dimensional submanifolds. On the other hand,
given the 2-mixed-coordinates of q(x), the projection p(x) ∈ Msbm of q(x) is proved to be
2 , 0, . . . , 0) (Amari et al., 1992). Thus, SBM defines a probabilistic parameter
[ζ]p = (ηi1 , ηij
space that is exactly derived from CIF.

4.2.2 SBM as Maximal Information Distance Projection
Next corollary, following Proposition 5, shows a geometric derivation of SBM. We make it
explicit that the projection on Msbm could maximally preserve the expected information
distance comparing to other tailored submanifolds of S with the same dimensionality k.
Corollary 6 Given the general manifold S in 2-mixed-coordinates [ζ]2 , SBM defines an kdimensional submanifold of S that can maximally preserve the expected information distance
induced by Fisher-Rao metric.
Proof in Appendix A.6.
From the CIF-based derivation, we can see that SBM confines the statistical manifold in
the parameter subspace spanned by those directions with high confidences, which is proved
to maximally preserve the expected information distance.
4. Both the two special BMs, i.e., SBM and RBM, can be trained using the ML method. Note that SBM
has no hidden units and hence no positive sampling is needed in training SBM.

11

4.2.3 The Relation Between [ζ]2t and the ML Learning of SBM
To learn such [ζ]2t , we need to learn the parameters ξ of SBM such that its stationary
distribution preserves the same coordinates [η 2− ] as target distribution q(x). Actually, this
is exactly what traditional gradient-based learning algorithms intend to do to train SBM.
Next proposition shows that the ML method for training SBM is equivalent to learn the
tailored 2-mixed coordinates [ζ]2t .
Proposition 7 Given the target distribution q(x) with 2-mixed coordinates:
2
[ζ]2 = (ηi1 , ηij
, θ2+ ),

the coordinates of the SBM with stationary distribution q(x; ξ), learnt by ML, are uniquely
2 ,θ
given by [ζ]2t = (ηi1 , ηij
2+ = 0)
Proof in Appendix A.7.

4.3 The CIF-based Derivation of Restricted Boltzmann Machine
In previous section, the general CIF uncovers why SBM uses the coordinates up to 2nd order, i.e., preserves the η-coordinates of the 1st -order and 2nd -order. In this section, we
will investigate the cases where hidden units are introduced. Particularly, one of the fundamental problem in neural network research is the unsupervised representation learning
(Bengio et al., 2013), which attempts to characterize the underlying distribution through
the discovery of a set of latent variables (or features). Many algorithmic models have been
proposed, such as restricted Boltzmann machine (RBM) (Hinton and Salakhutdinov, 2006)
and auto-encoders (Rifai et al., 2011; Vincent et al., 2010), for learning one level of feature
extraction. Then, in deep learning models, the representation learnt at one level is used as
input for learning the next level, etc. However, some important questions remain to be clarified: Do these algorithms implicitly learn about the whole density or only some aspects?
If they capture the essence of the target density, then can we formalize the link between the
essential part and omitted part? This section will try to answer these questions using CIF.
4.3.1 Two Necessary Conditions for Representation Learning
In terms of one level feature extraction, there are two main principles that guide a good
representation learning:
• Compactness of representation: minimize the redundancy between hidden variables
in the representation 5 .
• Completeness of reconstruction: the learnt representation captures sufficient information in the input, and could completely reconstruct input distribution, in a statistical
sense.
5. The concept of compactness in the neural network is of two-folds. 1) model-scale compactness: a
restriction on the number of hidden units in order to give a parsimonious representation w.r.t underlying
distribution; 2) structural compactness: a restriction on how hidden units are connected such that the
redundancy in the hidden representation is minimized. In this paper, we mainly focus on the latter case.

12

Let Sxh be the general manifold of probability distributions over the joint space of
visible units x and hidden units h, and Sx be the general manifold over visible units x.
Given any observation distribution q(x) ∈ Sx , our problem is to find the p(x, h) ∈ Sxh
with the marginal distribution p(x) that best approximates q(x), while p(x, h) is consistent
with the compactness and completeness conditions. Here, the K-L divergence, defined in
Equation (11), is used as the criterion of approximation.
First, we will investigate the submanifold of joint distributions p(x, h) ∈ Sxh that fulfill
the above two conditions. Let us denote this submanifold as Mcc . Extending Equation (2)
to manifold Sxh , p(x, h) has the θ-coordinates defined by:
X
θI XI − ψ
(18)
log p(x, h) =
I⊆{x,h}&I6=N ullSet

For the completeness requirement, it is easy to prove that the probability of any input
variable xi can be fully determined only by the given hidden representation and independent
with remainng input variables xj (j 6= i), if and only if θI = 0 for any I that contains two
or more input variables in Equation (18). Similarly, the compactness corresponds to the
extraction of statistically independent hidden variables given the input, i.e., θI = 0 for any
I that contains two or more hidden variables in Equation (18). Then Mcc is given by the
following coordinate system:
h

xh

xx

h hj

[θ]cc = (θ1xi , θ1 j , θ2 i j = 0, θ2 i j , θ2 i
| {z } |
{z
1−order

= 0, 0, . . . , 0).
} | {z }

(19)

orders>2

2−order

Then our problem is restated as to find the p(x, h) ∈ Mcc with the marginal distribution
p(x) that best approximates q(x).
4.3.2 The Equivalence between RBM and Mcc
RBM is a special kind of BM that restricts the interactions in Equation (12) only to those
between hidden and visible units, i.e., Uxi ,xj = 0, Vhi ,hj = 0 ∀xi , xj ∈ x; hi , hj ∈ h. Let
ξrbm = {W, b, d} denotes the set of parameters in RBM. Thus, the θ-coordinates for RBM
can be derived directly from Equation (15):
h

xx

xh

h hj

[θ]RBM = (θ1xi , θ1 j , θ2 i j = 0, θ2 i j , θ2 i
| {z } |
{z
1−order

2−order

= 0, 0, . . . , 0)
} | {z }

(20)

orders>2

Comparing Equation (19) to (20), the submanifold Mrbm defined by RBM is equivalent
with Mcc since they share exactly the same coordinate system. This indicates that the
compactness and completeness conditions is indeed realized by RBM. We use a simpler
notation B to denote Mrbm . Next, we will show how to use CIF to interpret the training
process of RBM.
4.3.3 The CIF-based Interpretation on the Learning of RBM
A RBM produces a stationary distribution p(x, h) ∈ Sxh over {x, h}. However, given the
target distribution q(x), only the marginal distribution of RBM over the visible units are
13

specified by q(x), leaving the distributions on hidden units vary freely. Let Hq be the
set of probability distributions q(x, h) ∈ Sxh that have the same marginal distribution
q(x) and the conditional distributions q(hj |x) of each hidden unit hj is realized by the
RBM’s activation function with parameter ξrbm (that is the logistic sigmoid activation:
1
f (hj |x; ξrbm ) = 1+exp{− P
Wij xi −dj } ):
i∈{1,...,nx }

Hq = {q(x, h) ∈ Sxh |∃ξrbm ,

X

q(x, h) = q(x), and q(h|x; ξrbm ) =

Y

f (hj |x; ξrbm )}

(21)

hj ∈h

h

Then our problem in Section 4.3.1 is restated with respect to Sxh : search for a RBM in B
that minimizes the divergence from Hq to B 6 .
Given p(x, h; ξp ), its best approximation on Hq is defined by the projection ΓH (p), which
gives the minimum K-L divergence from Hq to p(x, h; ξp ). Next proposition shows how the
projection ΓH (p) is obtained.
Proposition 8 Given a distribution p(x, h; ξp ) ∈ B, the projection ΓH (p) ∈ Hq that gives
the minimum divergence D(Hq , p(x, h; ξp )) from Hq to p(x, h; ξp ) is the q(x, h; ξq ) ∈ Hq that
satisfies ξq = ξp .
Proof in Appendix A.8.
On the other hand, given q(x, h; ξq ) ∈ Hq , the best approximation on B is the projection
ΓB (q) of q to B. In order to obtain an explicit expression of ΓB (q), we introduce the following
fractional mixed coordinates [ζ xh ] 7 for the general manifold Sxh :
hh

xx

[ζ xh ] = (ηx1i , ηh1j , θ2 i j , ηx2i hj , θ2 i j , θ2+ )
| {z } |
{z
} |{z}
1−order

2−order

(22)

orders>2

The [ζ xh ] is a valid coordinate system, that is, the relation between [θ] and [ζ xh ] is bijective.
This is shown in the next proposition.
Proposition 9 The relation between the two coordinate systems [θ] and [ζ xh ] is bijective.
Proof in Appendix A.9.
The next proposition gives an explicit expression of the coordinates for the projection
ΓB (q) learnt by RBM using the fractional mixed coordinates in Equation (22).
6. This restated problem directly follows from the fact that: the minimum divergence D(Hq , B) in the
whole manifold Sxh is equal to the minimum divergence D[q(x), Bx ] in the visible manifold Sx , shown
in Theorem 7 in Amari et al. (1992).
7. Note that both the fractional mixed coordinates [ζ xh ] and 2-mixed coordinates [ζ] are mixtures of ηcoordinates and θ-coordinates. In [ζ], coordinates of the same order are taken from either [η] or [θ].
h h
x x
However, in [ζ xh ], the 2nd -order coordinates consist of the {ηx2i hj } from [η] and {θ2 i j , θ2 i j } from [θ],
that is why the term “fractional” is used.

14

Figure 2: The iterative learning for RBM: in searching for the minimum divergence between
Hq and B, we first choose an initial RBM p0 and then perform projections ΓH (p)
and ΓB (q) iteratively, until the fixed points of the projections p∗ and q ∗ are
reached. With different initializations, the iterative projection algorithm may
end up with different local minima on Hq and B, respectively.

Proposition 10 Given q(x, h; ξq ) ∈ Hq with fractional mixed coordinates:
xx

hh

[ζ xh ]q = (ηx1i , ηh1j , θ2 i j , ηx2i hj , θ2 i j , θ2+ ),
the coordinates of the learnt projection ΓB (q) of q(x, h; ξq ) on the submanifold B are uniquely
given by:
hh
xx
(23)
[ζ xh ]ΓB (q) = (ηx1i , ηh1j , θ2 i j = 0, ηx2i hj , θ2 i j = 0, θ2+ = 0)
Proof This proof comes in three parts:
1. the projection ΓB (q) of q(x, h) on B is unique;
2. this unique projection ΓB (q) can be achieved by minimizing the divergence D[q(x, h), B]
using gradient descent method;
3. The fractional mixed coordinates of ΓB (q) is exactly the one given in Equation (23).
See Appendix A.10 for the detailed proof.
Back to the problem of obtaining the best approximation to the given target q(x), the
learning of RBM can be implemented by the following iterative projection process8 :
Let p0 (x, h; ξp0 ) be the initial RBM. For i = 0, 1, 2, . . . ,
1. Put qi+1 (x, h) = ΓH (pi (x, h; ξpi ))
8. Amari et al. (1992) proposed a similar iterative algorithm framework for the fully-connected BM. In
the present paper, we reformulate this iterative algorithm for the learning of RBM and give explicit
expressions of how the projections are achieved.

15

2. Put pi+1 (x, h; ξpi+1 ) = ΓB (qi+1 (x, h))
where ΓH (p) denotes the projection of p(x, h; ξp ) to Hq , and ΓB (q) denotes the projection
of q(x, h) to B. The iteration ends when we reach the fixed points of the projections p∗
and q ∗ , that is ΓH (p∗ ) = q ∗ and ΓB (q ∗ ) = p∗ . The iterative projection process of RBM is
illustrated in Figure 2. The convergence property of this iterative algorithm is guaranteed
using the following proposition:
Proposition 11 The monotonic relation holds in the iterative learning algorithm:
D[qi+1 , pi ] ≥ D[qi+1 , pi+1 ] ≥ D[qi+2 , pi+1 ], ∀i = {0, 1, 2, . . . }

(24)

where the equality holds only for the fixed points of the projections.
Proof in Appendix A.11.
The CIF-based iterative projection procedure (IP) for RBM gives us an alternative way
to investigate the learning process of RBM. The invariance in the learning of RBM is the
CIF: in the ith iteration, given qi ∈ Hq with fractional mixed coordinates:
hh

xx

[ζ xh ]qi = (ηx1i , ηh1j , θ2 i j , ηx2i hj , θ2 i j , θ2+ )
then the ordinates of the projection pi on B, i.e., ΓB (qi ), is given by Equation (23):
x xj

[ζ xh ]pi = (ηx1i , ηh1j , θ2 i

h hj

= 0, ηx2i hj , θ2 i

= 0, θ2+ = 0)

Now we will show that the process of the projection ΓB (qi ) can be derived from CIF, i.e.,
highly confident coordinates [ηx1i , ηh1j , ηx2i hj ] of qi are preserved while lowly confident coordinates [θ2+ ] are set to neutral value zero. For the fractional mix coordinates system
[ζ xh ], the closed form of its Fisher information matrix does not have the good expression
formula like Proposition 3 which are possessed by the mixed-coordinate system [ζ]. Next,
we will show that the fractional mix-coordinates of ΓB (qi ) can be derived in three steps by
jointly applying the CIF and the completeness and compactness conditions. Let the corresponding 2-mixed ζ-coordinates for qi be [ζ]2,qi = (ηx1i , ηh1j , ηx2i xj , ηx2i hj , ηh2i hj , θ2+ ). First,
we apply the general CIF for parametric reduction in [ζ]2,qi by replacing lowly confident
coordinates [θ2+ ] with neutral value zeros and preserving the remaining coordinates, resulting in the tailored mix-coordinates [ζ]2t ,qi = (ηx1i , ηh1j , ηx2i xj , ηx2i hj , ηh2i hj , θ2+ = 0), as
described in Section 3. Then, we transmit [ζ]2t ,qi into the fractional coordinate system, i.e.,
xx
hh
[ζ xh ]qi = (ηx1i , ηh1j , θ2 i j , ηx2i hj , θ2 i j , θ2+ = 0). Finally, the completeness and compactness
xx

hh

conditions require that the [θ2 i j , θ2 i j ] in [ζ xh ]qi are also set to neutral value zeros. Hence,
we can see that the coordinates of the projection ΓB (qi ) is exactly the one given by Equation
(23).
4.3.4 Comparison of The Iterative Projection and Gradient-based Methods
Given current parameters ξ i of RBM and samples x that generated from the underlying
distribution q(x), IP could be implemented in two phases:
16

1. In the first phase, we generate samples for the projection ΓH (pi ) of the stationary
distribution pi (x, h; ξ i ) on Hq . This is done by sampling h from RBM’s conditional
distribution pi (h|x; ξ i ) given x, and hence (x, h) ∼ ΓH (pi (x, h; ξ i ));
2. In the second phase, we train a new RBM with those generated samples (x, h), and
then update the RBM’s parameters to be the newly trained ones, denoted as ξ i+1 .
Note that in the second phase all hidden units in RBM are visible in samples (x, h). Thus
this sub-learning task is similar with training a BM without hidden units, which can be
implemented by traditional gradient-based methods.
Given current parameters ξ i of RBM (with stationary distribution pi ) and samples
x ∼ q(x), we can see that both ML and IP share the same sampling process, sampling
(x, h) in the ΓH (pi ) projection phase of IP and the positive phase of ML. In terms of the
quality of the sampling process, if x is sufficient and so is (x, h), both the ΓH (pi ) of IP and
positive phase of ML can achieve an accurate estimation of q(x, h; ξ i ). On the other hand,
if x is insufficient (it is usually true in real-world applications), sampling biases with respect
to q(x, h; ξ i ) may be introduced in the sampling process so that the accurate estimation can
not be guaranteed.
However, the updating rule is different: IP realizes the parameter updating by using a
sub-learning task (fitting a new RBM to the generated sample (x, h)), while ML adjusts the
parameters directly using Equation (17). Let qi+1 denote the distribution of (x, h), and pi+1
and p0i+1 denote the stationary distributions of RBM after the parameter updating using
IP and ML respectively. With a proper learning rate λ, the parameter updating phase of
ML would lead to the decrease of divergence, that is, D[qi+1 , pi ] ≥ D[qi+1 , p0i+1 ]. Since
pi+1 is the projection of qi+1 on B, then we have D[qi+1 , p0i+1 ] ≥ D[qi+1 , pi+1 ]. Therefore,
ML can be seen as an “unmature projection” of qi+1 on B, and it does not guarantee
that the theoretical projection ΓB (qi+1 ) is reached. To achieve the same projection point
ΓB (qi+1 ) as IP, ML needs multiple updating iterations, where each iteration moves the
current distribution towards ΓB (qi+1 ) in the gradient direction by some oracle step size
(controlled by the learning rate).
Another big difference is that IP separates the positive sampling process and the gradient
estimation in two phases: ΓH and ΓB , meaning that there is no positive sampling in the
sub-learning of ΓB . However, ML needs to constantly adjust the gradient direction with
respect to certain learning rate immediately after each sampling process. Later experiments
in Section 5.2 indicate that this may give IP the advantage of robustness against sampling
biases, especially when the gradient is too small to be distinguishable from these biases in
the learning process.
4.3.5 Discussions on deep Boltzmann machine
For deep Boltzmann machine (DBM) (Salakhutdinov and Hinton, 2012), several layers
of RBM compose a deep architecture in order to achieve a representation at a sufficient
abstraction level, where the hidden units are trained to capture the dependencies of units
at the lower layers, as shown in Figure 3. In this section, we give a discussion on some
theoretical insights on the deep architectures, in terms of the CIF principle.
In Section 4.3.2, we have shown that the structure of RBM implies the compactness and
completeness conditions, which could guide the learning of a good representation. Thus
17

Figure 3: A multi-layer BM with visible units x and hidden layers h(1) , h(2) and h(3) . The
greedy layer-wise training of deep architecture is to maximally preserve the confident information layer by layer. Note that the prohibition sign indicates that
the Fisher information on lowly confident coordinates is not preserved.

DBM can be seen as the composition of a series of representation learning stages. Then,
an immediate question is: what kind of representation of the data should be generated as
the output of each stage? From an information abstraction point of view, each stage of
the deep architecture could build up more abstract features by using the highly confident
information on parameters (or coordinates) that is transmitted from less abstract features in
lower layers. Those more abstract features would potentially have a greater representation
power (Bengio et al., 2013). The CIF principle describes how the information flows in those
representation transformations, as illustrated in Figure 3. We propose that each layer of
DBM determines a submanifold M of S, where M could maximally preserve the highly
confident information on parameters, as shown in Section 4.3.3. Then the whole DBM
can be seen as the process of repeatedly applying CIF in each layer, achieving the tradeoff
between the abstractness of representation features and the intrinsic information confidence
preserved on parameters.
Once a good representation has been found at each level by layer-wise application of
unsupervised greedy pre-training, it can be used to initialize and train the deep neural
networks through supervised learning (Hinton and Salakhutdinov, 2006; Erhan et al., 2010).
Recall that the straightforward application of gradient-based methods to train all layers of a
DBM simultaneously tends to fall into a poor local minima (Younes, 1998; Desjardins et al.,
2012). Now, our next question is: why can the layer-wise pre-training give a more reasonable
parameter initialisation? Empirically, Erhan et al. (2010) shows that the unsupervised pretraining acts as a regularization on parameters in a way that the parameters are set into a
region, from which better basins of attraction can be reached. Theoretically, by using the
fractional mixed coordinates, it can be shown that this regularized region is actually the
layer-wise restriction using CIF, i.e., the highly confident coordinates are preserved with
18

respect to the target density and all lowly confident coordinates are set to a neutral value of
zero, as illustrated in Figure 3. Effectively, the parameter space is regularized to fall into a
region where the parameters can be confidently estimated based on the given data. Under
this CIF-based regularization, the pre-training can be seen as searching for a reasonable
parameter setting, from which a good representation of the input data can be generated in
each layer.

5. Experimental Study
In this section, we will empirically investigate the CIF principle in density estimation tasks
on two types of Boltzmann machines, i.e., SBM and RBM. More specifically, for SBM, we
will investigate how to use CIF to take effect on the learning trajectory with respect to the
specific sample, and hence further confine the parameter space to the region corresponding to
the most confident information contained in given data. For RBM, it is inconvenient to use
a sample-specific strategy since the information of hidden variables is missed. Alternatively
we investigate the potential of the iterative projection procedure proposed previously. For
both SBM and RBM, two baseline learning methods, i.e., the contractive divergence (CD)
(Hinton, 2002; Carreira-Perpinan and Hinton, 2005) and maximum-likelihood (ML) (Ackley
et al., 1985), are adopted.
The ML learning is described in Section 4.1.2. The CD can be seen as an approximation
of ML. Let q(x) be the underlying probability distribution from which sample x are generated independently. Then our goal is to train a BM (with stationary probability p(x, h; ξ)
in Equation 13) based on x that realizes q(x) as faithfully as possible. Comparing to ML,
the CD learning realizes the gradient descend of a different objective function to avoid the
difficulty of computing the log-likelihood gradient in ML, shown as follows:
∆ξ = −ε ·

∂(D(q0 , p) − D(pm , p))
∂E(x, h; ξ)
∂E(x, h; ξ)
= ε · (−h
i0 + h
im )
∂ξ
∂ξ
∂ξ

(25)

where q0 is the sample distribution, pm is the distribution by starting the Markov chain
with the data and running m steps, h·i0 and h·im denote the averages with respect to the
distribution p0 and pm , and D(·, ·) denotes the K-L divergence.
In these experiments, two kinds of binary datasets are used:
1. The artificial binary dataset: we first randomly select the target distribution q(x),
which is chosen uniformly from the open probability simplex over the n random variables. Then, the dataset with N samples are generated from q(x).
2. The 20 News Groups binary dataset: 20 News Groups is a collection of approximately
20,000 newsgroup documents, partitioned evenly across 20 different newsgroups 9 . The
collection is preprocessed using porter stemmer and stop-word removal. We select top
100 terms with highest frequency in the collection. Each document is represented as
a 100-dimensional binary vector, where each element indicates whether certain term
occurs in current document or not.
9. The 20 News Group dataset is freely downloadable from http://qwone.com/∼jason/20Newsgroups/

19

5.1 Experiments with SBM
From the perspective of IG, we can see that ML/CD learning is to update parameters in
SBM so that its corresponding coordinates [η 2− ] are getting closer to the data distribution.
This is consistent with our theoretical analysis in Section 3 and Section 4.2 that SBM uses
the most confident information (i.e., [η 2− ]) for approximating an arbitrary distribution in
an expected sense. But, for the distribution with specific samples, can CIF further recognize
less-confident parameters in SBM and reduce them properly?
Our solution here is to apply CIF to take effect on the learning trajectory with respect to
specific samples, and hence further confine the parameter space to the region that indicated
by the most confident information contained in the samples. This experiment shows that
given specific samples we need to preserve the confident parameters to certain extend, and
there should exist some golden ratio that would produce best performance on average.
5.1.1 A Sample-specific CIF-based CD Learning
The main modification of our CIF-based CD algorithm (CD-CIF for short) is that we
generate the samples for pm (x) based on those parameters with confident information,
where the confident information carried by certain parameter is inherited from the sample
and could be assessed using its Fisher information computed in terms of the sample.
For CD-1 (i.e., m=1), the firing probability for the ith neuron after one-step transition
(0) (0)
(0)
from the initial state x(0) = {x1 , x2 , . . . , xn }) is:
(1)

p(m=1) (xi

1

= 1|x(0) ) =
1 + exp{−

(0)
j6=i Uij xj

P

(26)
− bi }

For CD-CIF, the firing probability in Equation 26 is modified as follows:
1
(1)
p0(m=1) (xi = 1|x(0) ) =
P
(0)
1 + exp{− (j6=i)&(F (Uij )>τ ) Uij xj − bi }

(27)

where τ is a pre-selected threshold, F (Uij ) = Eq0 [xi xj ]−Eq0 [xi xj ]2 is the Fisher information
of Uij (see Equation 8) and the expectations are estimated from the given sample x. We
can see that those weights whose Fisher information are less than τ are considered to be
unreliable w.r.t. x. In practice, we could setup τ by the ratio r to specify the remaining
proportion of the total Fisher information TF I of all parameters, i.e., τ = r ∗ TF I .
In summary, CD-CIF is realized in two phases. In the first phase, we initially “guess”
whether certain parameter could be faithfully estimated based on the finite sample. In the
second phase, we approximate the gradient using the CD scheme, except for the CIF-based
firing function is used.
5.1.2 Results and Discussions on Artificial Dataset
In this section, we empirically investigate our justifications for the CIF principle, especially
how the sample-specific CIF-based CD learning works in the context of density estimation.
Experimental Setup and Evaluation Metric: For computation simplicity, the artificial dataset is set to be 10-dimensional. Three learning algorithms are investigated: ML,
CD-1 and our CD-CIF. K-L divergence is used to evaluate the goodness-of-fit of the SBM
20

Density Estimation Performance for CD−CIF

Performance Change with varying amount of Confident Information (1200)
3

Performance Change with varying amount of Confident Information (100)

0.55

0.4
0.35
0.3
0.25
0.2

0

200

400
600
800
Number of samples N

(a)

1000

1200

CD−1
ML
CD−CIF

0.52

K−L Divergence (Background vs SBMs)

0.45

K−L Divergence (Background vs SBMs)

K−L Divergence (Background vs SBMs)

0.54
CD−1
ML
CD−CIF

0.5

0.5
0.48
0.46
0.44
0.42
0.4
0.38
0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Learning Trajectory using 400 samples
0.36

True Distribution

0.35
0.34

2

0.33

0.14

1.5

0.135
1

0.13
0.94

0.32
0.96

0.98

1

0.31

0.5

0
0.1

1

CD−1
ML
CD−CIF

2.5

Fixed Point
of CD−CIF

Fixed Point of CD−1

Fixed Point of ML
Training Start Point

0.3

0.2

0.3

0.4

0.5

0.6

r

r

(b)

(c)

0.7

0.8

0.9

1

0.29
0.93

0.94

0.95

0.96

0.97

0.98

0.99

(d)

Figure 4: (a): the performances of CD-CIF on different sample sizes; (b) and (c): the
performances of CD-CIF with various values of r on two typical sample sizes, i.e.,
100 and 1200; (d): learning trajectories of last 100 steps for ML (squares), CD-1
(triangles) and CD-CIF (circles).

trained by various algorithms. For sample size N , we run 100 instances (20 randomly generated distributions × 5 randomly running) and report the averaged K-L divergences. Note
that we focus on the case that the variable number is relatively small (n = 10) in order to
analytically evaluate the K-L divergence and give a detailed study on algorithms. Changing the number of variables only offers a trivial influence for experimental results since we
obtained qualitatively similar observations on various variable numbers (not reported here).
Automatically Adjusting r for Different Sample Sizes: The Fisher information
is additive for i.i.d. sampling. When sample size N changes, it is naturally to require that
the total amount of Fisher information contained in all tailored parameters is steady. Hence
we have α = (1 − r)N , where α indicates the amount of Fisher information and becomes a
constant when the learning model and the underlying distribution family are given. It turns
out that we can first identify α using the optimal r w.r.t. several distributions generated
from the underlying distribution family, and then determine the optimal r for various sample
sizes using: r = 1 − α/N . In our experiments, we set α = 35.
Density Estimation Performance: The averaged K-L divergences between SBM
(learned by ML, CD-1 and CD-CIF with the r automatically determined) and the underlying
distribution are shown in Figure 4(a). In the case of relatively small samples (N ≤ 500)
in Figure 4(a), our CD-CIF method shows significant improvements over ML (from 10.3%
to 16.0%) and CD-1(from 11.0% to 21.0%). This is because we could not expect to have
reliable identifications for all model parameters from insufficient samples, and hence CD-CIF
gains its advantages by using parameters that could be confidently estimated. This result is
consistent with our previous theoretical insight that Fisher information gives a reasonable
guidance for parametric reduction via the confidence criterion. As the sample size increases
(N ≥ 600), CD-CIF, ML and CD-1 tend to have similar performances. Since with relatively
large samples most model parameters can be reasonably estimated, and hence the effect of
parameter reduction using CIF gradually becomes marginal. In Figure 4(b), Figure 4(c), we
show how sample size affects the interval of r that achieves improvements over CD-1. For
N = 100, CD-CIF achieves significantly better performances for a wide range of r. While,
for N = 1200, CD-CIF can only marginally outperform baselines for a narrow range of r.
Effects on Learning Trajectories: We use the 2D visualizing technology SNE to
investigate learning trajectories and dynamical behaviors of three comparative algorithms
21

Density estimation performance on 20 News Groups
22
21.5

Hamming distance

21
20.5

CD−1
ML
CD−CIF

20
19.5
19
18.5
18
0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

r

Figure 5: the performances of CD-CIF with various values of r on 20 news group dataset
(Carreira-Perpinan and Hinton, 2005). We start three methods with the same parameter initialization. Then each intermediate state is represented by a 55-dimensional vector
formed by its current parameter values. From Figure 4(d), we can see that: 1) In the final
100 steps, three methods seem to end up with staying in different regions of the parameter
space, and CD-CIF confines the parameter in a relatively thinner region compared to ML
and CD-1; 2) The true distribution is usually located on the side of CD-CIF, indicating its
potential for converging to the optimal solution. Note that the above claims are based on
general observations and Figure 4(d) is shown as an illustration. Hence we may conclude
that CD-CIF regularizes the learning trajectories in a desired region of the parameter space
using the sample-specific CIF principle.
5.1.3 Results and Discussions on Real Textual Dataset
In this section, we empirically investigate how the sample-specific CIF-based CD learning
works on real-world datasets in the context of density estimation. In particular, we use
the SBM to learn the underlying probability density over 100 terms of the 20 News Groups
binary dataset. The learning rate for CD-1, ML and CD-CIF are manually tuned in order to
converge properly and all set to 0.001. Since it is infeasible to compute the K-L devergence
due to the high dimensionality, the averaged Hamming distance between the samples in the
dataset and those generated from the SBM is used to evaluate the goodness-of-fit of the
SBM’s trained by various algorithms. Let D = {d1 , d2 , . . . , dN } denote the dataset of N
documents, where each document di is a 100-dimensional binary vector. To evaluate a SBM
with parameter ξsbm , we first randomly generate N samples from the stationary distribution
p(x; ξsbm ), denoted as V = {v1 , v2 , . . . , vN }. Then the averaged hamming distance Dham is
calculated as follows:
P
di (minvj (Ham[di , vj ])
Dham [D, V ] =
N
where Ham[di , vj ] is the number of positions at which the corresponding values are different.
22

The result is shown in Figure 5. Our CD-CIF method shows maximal improvements
over ML (12.15%) and CD-1(15.05%) at r = 0.92. We can also see that CD-CIF achieves
significantly better performances for a wide range of r ∈ [0.5, 0.96], which is consistent with
our observations with the experiments on artificial datasets when the samples is insufficient.
5.2 Experiments with RBM
The RBM is practically more interesting than SBM, since it has higher representational
power. In this section, we will compare three different learning algorithms for RBM: CD-1,
ML and IP. In Carreira-Perpinan and Hinton (2005), CD is shown to be biased with respect
to ML for almost all data distributions. In Section 4.3.4, we have compared ML and IP
theoretically. In this section, an empirical study on the three algorithms is conducted.
5.2.1 Results and Discussion on Artificial Datasets
Experimental Setup and Evaluation Metric: For computational simplicity, the artificial dataset is of 5 dimensionality, and the number of hidden units in RBM is set to
5. Three learning algorithms are investigated: ML, CD-1 and IP. K-L divergence is used
to evaluate the goodness-of-fit of the RBM’s trained by various algorithms. Six different
sample sizes N are tested, namely 50, 100, 200, 500, 5000 and 50000. For sample size N ,
the learning rates for CD-1 and ML are set to be ε = 0.5/N , and we observe that they
could converge properly. For the sub-learning phase ΓB of IP, we adopt the CD algorithm
for the training of BM without hidden units, whose learning rate is also set to ε = 0.5/N .
We need to scan the dataset multiple times in order to iteratively update parameters, and
each full scan of the whole dataset is called an epoch. In CD-1 and ML, we set the maximal
number of epoches to 8000. We run the IP for 40 iterations, and each iteration is a CD
sub-training with the maximal number of epoches setting to 200. We adopt CD-1 as the
baseline method.
Results and Analysis: The on-average performances of the three methods on dataset
of different scales are shown in Table 1 in the context of density estimation. In order to study
the behavior of IP, we plot the sequences of K-L divergence between target distribution and
RBM in each iteration along the whole learning trajectory, shown in Figure 6. Comparing
CD-1 with ML, we can see that the K-L divergences of both CD-1 and ML decrease in a
similar way, converging at the same rate, taking the same number of iterations to converge
to a given tolerance, which is consistent with the conclusion in Hinton (2002).
From the convergence behavior of IP shown in Figure 6, we can see that the general trend
is that the K-L divergence decreases steadily with small fluctuations. Since there are only
few iterations in IP, it is reasonable for us to select the best performance that IP has reached
in the whole learning process, called the best IP. Here the K-L divergence between the sample
distribution and RBM’s stationary distribution is adopted as the selection metric. Thus, in
addition to the converging performance, we also show the best performance selected among
all 40 iterations for IP in Table 1. Note that the best performances for CD-1 and ML are
not reported, since their converging performances are often approximately the best ones.
For small sample size (e.g., 50 and 100), we can see that the converging performance of
IP is comparable with respect to CD and ML. As the sample size increases, we can see that
IP gradually outperforms CD-1 and ML, and shows significant improvement on large sample
23

sample size = 50000

sample size = 100

1.4

1.4
ML
CD−1
IP

1.2

1.2

1
K−L divergence

1
K−L divergence

ML
CD−1
IP

0.8

0.6
epoch=200
epoch=2000

0.8

0.6

0.4

0.4

epoch=200

epoch=4000

epoch=4000
0.2

0.2

0

epoch=2000

0

0.5

1

1.5

2

2.5

3

3.5

0

4

0

0.5

1

1.5

2

Iteration 10x

Iteration 10x

(a)

(b)

2.5

3

3.5

4

Figure 6: (a) and (b) illustrate the averaged learning curves for CD-1, ML and IP for 30
randomly chosen data distributions, with sample size 100 and 50000 respectively.
The x-axis is in log10 scale. To compare the efficiency of different algorithms
along the time-line (the time unit is an epoch, that is, the time used for the
parameter updates in each full scan of the whole dataset), we plot three time
stamps (epoch=100, 1000 and 4000). Note that each iteration in IP contains a
sub-training task, which is trained using 200 epoch in this experiment.

size (e.g., 5000 and 50000). For the best IP, its performance is significantly better than CD1 and ML for all sample sizes, indicating that IP has the potential to further improve its
performance by using some suitable selection metric. Comparing to ML, IP takes much
shorter iterations as expected to achieve a performance threshold. Theoretically, IP can
converge to the local minimum of RBM based on our theoretical analysis in Section 4.3.4.
If a proper learning rate is selected, ML can also converge to a local minimum. However,
one interesting result is that sometimes there is a big difference between the convergence
points of IP and ML, even in cases that the sampling is sufficient (e.g., sample size equals
50000).
This can be explained as follows. In practice, ML needs to constantly do positive
and negative sampling in each updating, which may produce much sample biases. As the
gradient decreasing to a small value, the correct gradient direction may be fluctuated by
the sample biases. Thus, instead of converging to the local minimum, ML might fluctuate
around some sub-optimal region. Actually, the ΓH in IP is a sampling process that may also
introduce sample biases. That is why the IP is fluctuated when the sample is insufficient, as
shown in Figure 6(a). Given sufficient samples, the sampling biases in ΓH decreases and the
fluctuation of IP declines, as shown in Figure 6(b). For ML, though the sampling biases for
the whole dataset becomes small given sufficient samples, the gradient estimation for each
sample is still closely intergraded with the sample bias of certain sample, meaning that
this inseparable coupling relationship results in a biased gradient estimation. The main
advantage of the IP procedure over traditional gradient-based methods is the separation
24

Sample Size
50
100
200
500
5000
50000

CD-1
0.0970
0.0786
0.0672
0.0621
0.0532
0.0497

ML
0.0971
0.0793
0.0678
0.0621
0.0524
0.0475

Converge IP (-chg%)
0.0978 (+0.81%)
0.0787 (+0.08%)
0.0640 (-4.75%)
0.0594 (-4.25%)
0.0468 (-12.00%)
0.0411 (-17.37%)

Best IP (-chg%)
0.0774 (-20.21%)
0.0637 (-18.92%)
0.0575 (-14.45%)
0.0567 (-8.61%)
0.0437 (-17.81%)
0.0332 (-33.08%)

Table 1: Performance comparison of CD-1, ML and IP in the density estimation task. The
change of IP with respect to the baseline method (CD-1) is reported.

Density estimation performance for 20 News Groups
18

Hamming distance

17
CD−1
ML
IP

16

15

14

13

12
10

20

30

40
50
60
70
Number of hidden units

80

90

100

Figure 7: Performances of IP with various number of hidden units on 20 News Group
of the positive sampling process and the gradient estimation. We conjecture that this
independent design gives IP the potential to achieve optimal solutions that CD-1 and ML
can not reach.
5.2.2 Results and Discussion on Real Textual Dataset
In this section, we empirically investigate how the IP works for RBM on real-world datasets
in the context of density estimation. We use the RBM to learn the probability density
over 100 terms on the 20 News Groups binary dataset. The number of hidden units nh in
RBM is set to [10, 20, . . . , 100]. The learning rate for CD-1and ML are manually tuned in
order to converge properly and all set to 0.001. The learning rate for the CD sub-learning
task in IP is also set to 0.001. We run IP for 9 iterations for all settings of nh . Similar
with the experiments in Section 5.1.3, the averaged Hamming distance is used to evaluate
the goodness-of-fit of the RBM’s trained by various algorithms. The average Hamming
distances for ML, CD-1 and IP are shown in Figure 7. We can see that IP achieves better
25

performances on all settings of nh . And the hamming distance for IP drops dramatically
as nh increases. This trend can be explained as follows. As nh grows, the sampling biases
increases and the interference of sampling biases with respect to the gradient estimation
becomes more and more serious. This limits the actual performance of RBMs learnt by
CD-1 and ML, with respect to the growing modelling power gained by increasing nh . As
shown in Section 5.2.1, the IP procedure separates the positive sampling process and the
gradient estimation in two phases: ΓH and ΓB . This result shows that IP has the potential
to achieve optimal solutions that CD-1 and ML can not reach in real-world applications.

6. Conclusions
The CIF principle proposed in this paper tackles the problem of dimensionality reduction in
parameter space by preserving the parameters with highly confident estimates and tailoring the less confident parameters. It provides a strategy for the derivation of probabilistic
models. The SBM and RBM are specific examples in this regard. They have been theoretically shown to achieve a reliable representation in parameter spaces by exactly using the
general CIF principle. CIF gives us a principled and context-independent way to address
the questions on what we should do for parameter reduction (regularization) and how to do
it. Based on CIF, we also show that the deep neural networks consisting of several layers of
RBM can be seen as the layer-wise usage of CIF, leading to some theoretical interpretations
of the rationale behind deep learning models.
One interesting result shown in our experiments is that: although CD-CIF is a biased algorithm, it could significantly outperform ML when the sample is insufficient. This suggests
that CIF gives us a reasonable criterion for recognizing and utilizing confident information
from the underlying data while ML fails to do so. Another interesting observation is that
ML and the CIF-based IP lead to different convergence points in the training of RBM.
Our experimental results indicate that IP has the advantage of robustness against sampling
biases, due to the separation of the positive sampling process and the gradient estimation.
In the future, we will further develop the formal justification of CIF w.r.t various contexts
(e.g., distribution families or models). We will also conduct more extensive experiments on
real world applications, such as document classification and handwritten digit recognition,
to further justify the properties of IP. We will also extend the IP to train deep neural
networks.

Acknowledgments
This work is partially supported by the Chinese National Program on Key Basic Research
Project (973 Program, grant no. 2013CB329304 and 2014CB744604), the Natural Science
Foundation of China (grants no. 61070044, 61111130190, 61272265 and 61105072), and
the European Union Framework 7 Marie-Curie International Research Staff Exchange Programme (grant no. 247590).
26

Appendix A.
A.1 Proof of Proposition 1
Proof By definition, we have:
gIJ =

∂ 2 ψ(θ)
∂θI ∂θJ

where ψ(θ) is defined by Equation (4). Hence, we have:
gIJ

P
∂ 2 ( I θI ηI − φ(η))
∂ηI
= J
=
∂θI ∂θJ
∂θ

By differentiating ηI , defined by Equation (1), with respect to θJ , we have:
gIJ

P
P
∂ x XI (x)(exp{ I θI XI (x) − ψ(θ)})
∂ηI
=
=
∂θJ
∂θJ
X
=
XI (x)[XJ (x) − ηJ ]p(x; θ) = ηI S J − ηI ηJ
x

This completes the proof.
A.2 Proof of Proposition 2
Proof By definition, we have:
g IJ =

∂ 2 φ(η)
∂ηI ∂ηJ

where φ(η) is defined by Equation (4). Hence we have:
g

IJ

=

P
∂ 2 ( J θJ ηJ − ψ(θ))
∂θI
=
∂ηI ∂ηJ
∂ηJ

Based on Equation (2) and (1), the θI and pK could be calculated by solving a linear
equation system of [p] and [η] respectively. Hence we have:
θI =

X

X

(−1)|I−K| log(pK ); pK =

K⊆I

(−1)|J−K| ηJ

K⊆J

Therefore, the partial derivation of θI with respect to ηJ is:
g IJ =

∂θI
∂ηJ

=

X ∂θI ∂pK
X
1
(−1)|I−K|+|J−K| ·
·
=
∂pK ∂ηJ
pK
K

K⊆I∩J

This completes the proof.

27

A.3 Proof of Proposition 3
Proof
The

 Fisher information matrix of [ζ] could be partitioned into four parts: Gζ =
A C
. It can be verified that in the mixed coordinate, the θ-coordinate of order k
D B
is orthogonal to any η-coordinate less than k-order, impling the corresponding element of
Fisher information matrix is zero (C = D = 0) (Nakahara and Amari, 2002). Hence, Gζ is
a block diagonal matrix.
According to Cramér-Rao bound (Rao, 1945), a parameter (or a pair of parameters)
has a unique asymptotically tight lower bound of the variance (or covariance) of unbiased
estimate, which is given by the corresponding element of the inverse of Fisher information
matrix involving this parameter (or this pair of parameters). Recall that Iη is the index
set of the parameters shared by [η] and [ζ]l and Jθ is the index set of the parameters
−1
−1
−1
−1
=
shared by [θ] and [ζ]l , we have (G−1
ζ )Iζ = (Gη )Iη and (Gζ )Jζ = (Gθ )Jθ , i.e., Gζ


−1
(Gη )Iη
0
. Since Gζ is a block tridiagonal matrix, the proposition follows.
0
(G−1
θ )Jθ
A.4 Proof of Proposition 4

U X
, which is parXT V
titioned based on Iη and Jθ . Based on Proposition 3, we have A = U −1 . Obviously, the
diagonal elements of U are all smaller than one. According to the succeeding Lemma 12,
we can see that the diagonal elements of A (i.e., U −1 ) are greater than 1.
Next we need to show that the diagonal elements of B are smaller than 1. Using the
−1
Schur complement of Gθ , the bottom-right block of G−1
θ , i.e., (Gθ )Jθ , equals to (V −
X T U −1 X)−1 . Thus the diagonal elements of B: Bjj = (V − X T U −1 X)jj < Vjj < 1. Hence
we complete the proof.


Proof Assume the Fisher information matrix of [θ] be Gθ =

Lemma 12 With a l × l positive definite matrix H, if Hii < 1, then (H −1 )ii > 1, ∀i ∈
{1, 2, . . . , l}.
Proof Since H is positive definite, it is a Gramian matrix of l linearly independent vectors
v1 , v2 , . . . , vl , i.e., Hij = hvi , vj i (h·, ·i denotes the inner product). Similarly, H −1 is the
Gramian matrix of l linearly independent vectors w1 , w2 , . . . , wl and (H −1 )ij = hwi , wj i.
It is easy to verify
that hwi , vi i = 1, ∀i ∈ {1, 2, . . . , l}. If Hii < 1, we can see that the
√
norm kvi k = Hii < 1. Since kwi k × kvi k ≥ hwi , vi i = 1, we have kwi k > 1. Hence,
(H −1 )ii = hwi , wi i = kwi k2 > 1.
A.5 Proof of Proposition 5
Proof Let Bq be a ε-ball surface centered at q(x) on manifold S, i.e., Bq = {ζ ∈ S|kζ −
ζq k2 = ε}, where k · k2 denotes the Euclid norm and ζq is the coordinates of q(x). Let
q(x) + dq be a neighbor of q(x) uniformly sampled on Bq and ζq(x)+dq be its corresponding
coordinates. For a small ε, we can calculate the expected information distance between
q(x) and q(x) + dq as follows:
Z
1
EBq = [(ζq(x)+dq − ζq )T Gζ (ζq(x)+dq − ζq )] 2 dBq
(28)
28

where Gζ is the Fisher information matrix at q(x).
Since Fisher information matrix Gζ is both positive definite and symmetric, there exists
a singular value decomposition Gζ = U T ΛU where U is an orthogonal matrix and Λ is a
diagonal matrix with diagonal entries equal to the eigenvalues of Gζ (all ≥ 0).
Apply the singular value decomposition into Equation (28), the distance becomes:
Z
1
EBq = [(ζq(x)+dq − ζq )T U T ΛU (ζq(x)+dq − ζq )] 2 dBq
(29)
Note that U is an orthogonal matrix, and the transformation U (ζq(x)+dq − ζq ) is a normpreserving rotation.
Now we need to show that among all tailored k-dimensional submanifolds of S, [ζ]lt is
the one that preserves maximum information distance. Assume IT = {i1 , i2 , . . . , ik } is the
index of k coordinates that we choose to form the tailored submanifold T in the mixedcoordinates [ζ]. Based on Equation (29), the expected information distance EBq for T is
proportional to the sum of eigenvalues of the sub-matrix (Gζ )IT , where the sum equals to
the trace of (Gζ )IT .
Next we show that the sub-matrix of Gζ specified by [ζ]lt gives maximum trace. Based
on Proposition 4, the elements on the main diagonal of the sub-matrix A are lower bounded
by one, and those of B upper bounded by one. Therefore, [ζ]lt gives maximum trace among
all sub-matrices of Gζ . This completes the proof.
A.6 Proof of Proposition 6
Proof Let Msbm be the set of all probability distributions realized by SBM. Amari
et al. (1992) proves that the mixed-coordinates of the resulting projection P on Msbm
2 , 0, . . . , 0), given the 2-mixed-coordinates of q(x). M
is [ζ]P = (ηi1 , ηij
sbm is equivalent to the
submanifold tailored by CIF, i.e. [ζ]2t . The corollary follows from Proposition 5.
A.7 Proof of Proposition 7
Proof Based on Equation 15, the coordinates [θ2+ ] for SBM is zero: θ2+ = 0. Next, we
2 ] with q(x).
show that the stationary distribution p(x; ξ) learnt by ML has the same [ηi1 , ηij
For SBM, the

∂E(x;ξ)
∂ξ

can be easily calculated from Equation (12):

 ∂E(x;ξ) = xi xj ,
∂Ux x

f or Uxi xj ∈ ξ;

 ∂E(x;ξ)
∂bxi

f or bxi ∈ ξ.

i j

= xi ,

Thus, based on Equation 17, the gradients for Uxi xj , bxi ∈ ξ are as follows:

 ∂ log p(x;ξ) = hxi xj i0 − hxi xj i∞ = η 2 (q(x)) − η 2 (p(x; ξ)), f or Uxi xj ∈ ξ;
ij
ij
∂Ux ,x
i

j

 ∂ log p(x;ξ) = hxi i0 − hxi i∞ = ηi1 (q(x)) − ηi1 (p(x; ξ)),
∂bx
i

f or bxi ∈ ξ.

where h·i0 denotes the average using the sample data and h·i∞ denotes the average with
respect to the stationary distribution p(x; ξ).
29

Since SBM defines an e-flat submanifold Msbm of S (Amari et al., 1992), then ML
converges to the unique solution that gives the best approximation p(x; ξ) ∈ Msbm of q(x).
p(x;ξ)
When ML converges, we have ∆ξ → 0 and hence ∂ log∂ξ
→ 0. Thus, we can see that ML
2 ] of q(x). This
converges to stationary distribution p(x; ξ) that preserves coordinates [ηi1 , ηij
completes the proof.
A.8 Proof of Proposition 8
Proof Based on the definition of divergence in Equation (11), the following relation holds:
D[q(x, h), p(x, h)] = D[q(x)q(h|x), p(x)p(h|x)]
q(x)
q(h|x)
= Eq(x,h) [log
+ log
]
p(x)
p(h|x)
= D[q(x), p(x)] + Eq(x) [D[q(h|x), p(h|x)]]
where Eq(x,h) [·] and Eq(x) [·] are the expectations taken over q(x, h) and q(x) respectively.
Therefore, the minimum divergence between p(x, h; ξp ) and Hq is given as:
D(Hq , p(x, h; ξp )) =

min
q(x,h;ξq )∈Hq

D[q(x, h; ξq ), p(x, h; ξp )]

= min{D[q(x), p(x)] + Eq (x)[D[q(h|x; ξq ), p(h|x; ξp )]]}
ξq

= D[q(x), p(x)] + min{Eq(x) [D[q(h|x; ξq ), p(h|x; ξp )]]}
ξq

= D[q(x), p(x)]
In the last equality, the expected divergence between q(h|x; ξq ) and p(h|x; ξp ) vanishes if
and only if ξq = ξp . This completes the proof.10
A.9 Proof of Proposition 9
Proof First, we show that [ζ xh ] is determined given [θ]. Since there is a one-to-one
correspondence between coordinates [θ] and [p], [ζ xh ] can be directly calculated from the
p-coordinates corresponding to [θ] based on Equation (1) and (2).
hh
xx
Second, [θ] is determined by knowing [ζ xh ]. The {θ2 i j , θ2 i j , θ2+ } part of [θ] are set to be
xx
hh
h
xh
equal to those in [ζ xh ]. By fixing {θ2 i j , θ2 i j , θ2+ } and setting {θ1xi , θ1 j , θ2 i j } free, we now
have an e-flat smooth submanifold R. Assume that there exist two different distributions
P1 and P2 with coordinates [θ]1 and [θ]2 that have the same mixed coordinates [ζ xh ]. Thus
both P1 and P2 belong to R and share the same value of {ηx1i , ηh1j , ηx2i hj }. Let Q ∈ Sxh be a
distribution whose projection on R is P1 . Based on the Projection Theorem in Amari and
Nagaoka (1993), P1 is the unique closest point on R to Q. Considering the minimization of
the divergence D[Q, PR ] between PR ∈ R and Q, the gradient vector of D[Q, PR ] over the
h
xh
free parameters {θ1xi , θ1 j , θ2 i j } at P1 , that is {ηx1i , ηh1j , ηx2i hj }PR −{ηx1i , ηh1j , ηx2i hj }Q , equals to
zero vector. Then, P2 also has a zero-gradient vector and hence is the projection point of Q,
since P2 has the same {ηx1i , ηh1j , ηx2i hj } with P1 . However, since R is e-flat 11 , the projection
10. Note that a similar path of proof is also used in Theorem 7 of Amari et al. (1992), which is for the
fully-connected BM. Here, Here, we reformulate the proof for RBM to derive the projection ΓH (p(x, h)).
11. For more information about the concept of flatness, please refer to the book Amari and Nagaoka (1993).

30

of Q on R is unique, meaning that P1 and P2 are the same point. Therefore, there does
not exist two different distributions P1 and P2 that have the same mixed coordinates [ζ xh ].
This completes the proof.
A.10 Proof of Proposition 10
Proof First, we prove the uniqueness of the projection ΓB (q). From the [θ] of RBM in
Equation (20), B is an e-flat smooth submanifold of Sxh . Thus the projection is unique.
Note that
Second, in order to find the p(x, h; ξp ) ∈ B with parameter ξp that minimizes the
divergence between q(x, h; ξq ) and B, the gradient descent method iteratively adjusts ξp
in the negative gradient direction that the divergence D[q, p(ξp )] decreases fastest:
4ξp = −λ

∂D[q, p(ξp )]
∂ξp

where D[q, p(ξp )] is treated as a function of RBM’s parameters ξp and λ is the learning rate.
As shown in Albizuri et al. (1995), the gradient descent method converges to the minimum
of the divergence with proper choices of λ, and hence achieves the projection point ΓB (q).
Last, we show that the fractional mixed coordinates [ζ xh ]ΓB (q) in Equation (23) is exactly
the convergence point of the learning for RBM . We calculate the first-order derivative of
D[q, p(ξp )], w.r.t ξp , where p(x, h; ξp ) = Z1 exp{−E(x, h; ξp )} is given in Equation (13).
For Wxi ,hj in ξp (denoted as Wij ), we have:
X q(x, h) ∂p(x, h; ξp )
∂D[q, p(ξp )]
=−
∂Wij
p(x, h; ξp )
∂Wij

(30)

x,h

where the

∂p(x,h)
∂Wij

is calculated as follows:

∂p(x, h; ξp )
∂Wij

= Z −1 exp{−E(x, h)}{

∂(−E(x, h)) X
∂(−E(x, h))
−
p(x, h)
}
∂Wij
∂Wij
x,h

∂(−E(x, h))
∂(−E(x, h))
− p(x, h)
p(x, h) ·
∂Wij
∂Wij
x,h
X
= p(x, h) · xi hj − p(x, h) ·
p(x, h)xi hj
X

= p(x, h) ·

(31)

x,h

Combining Equation (30) and (31), we have:
X
X
∂D[q, p(ξp )]
=−
q(x, h)xi hj +
p(x, h)xi hj = ηx2i hj (p) − ηx2i hj (q)
∂Wij
x,h

(32)

x,h

where ηx2i hj (p) and ηx2i hj (q) denotes the 2nd -order η-coordinates of p and q respectively.
Similarly, the first-order derivatives for biases bxi and dhj can be proved to be:
∂D[q, p(ξp )]
= ηx1i (p) − ηx1i (q)
∂bxi
31

(33)

∂D[q, p(ξp )]
= ηh1j (p) − ηh1j (q)
∂dhj

(34)

Summarizing Equation (32), (33) and (34), the first-order derivatives of ξpI , where the
indexing I = {xi } or {hj } or {xi , hj } (∀ xi ∈ x, hj ∈ h), can be calculated in the same
way, that is subtracting p’s and q’s corresponding η-coordinates ηI :
∂D[q, p(ξp )]
= ηI (p) − ηI (q)
∂ξpI
When converging, we have

∂D[q,p(ξp )]
∂ξpI

→ 0. Hence, the gradient descent method con-

verges to the projection point ΓB (q) with a stationary distribution p(x, h; ξp ) that preserves
coordinates [ηx1i , ηh1j , ηx2i hj ] of q(x, h; ξq ). This completes the proof.
A.11 Proof of Proposition 11
Proof Since pi ∈ B and pi+1 ∈ B is the projection of qi+1 , then D[qi+1 , pi ] ≥ D[qi+1 , pi+1 ].
Similarly, qi+1 ∈ Hq and qi+2 ∈ Hq is the projection of pi+1 , thus D[qi+1 , pi+1 ] ≥ D[qi+2 , pi+1 ].
This completes the proof.

References
D. H. Ackley, G. E. Hinton, and T. J. Sejnowski. A learning algorithm for Boltzmann
machines. Cognitive Science, 9:147–169, 1985.
F. X. Albizuri, A. d’Anjou, M. Grana, J. Torrealdea, and M. C. Hernandez. The highorder boltzmann machine: learned distribution and topology. Neural Networks, IEEE
Transactions on, 6(3):767–770, 1995.
S. Amari and H. Nagaoka. Methods of Information Geometry. Translations of Mathematical
Monographs. Oxford University Press, 1993.
S. Amari, K. Kurata, and H. Nagaoka. Information geometry of boltzmann machines. IEEE
Transactions on Neural Networks, 3(2):260–271, 1992.
Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy layer-wise training of deep
networks. In NIPS’06, pages 153–160, Vancouver, British Columbia, Canada, 2006.
Y. Bengio, A. C. Courville, and P. Vincent. Representation learning: A review and new
perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798–1828, 2013.
M. A. Carreira-Perpinan and G. E. Hinton. On contrastive divergence learning. Artificial
Intelligence and Statistics, pages 17–24, 2005.
N. N. Chentsov. Statistical Decision Rules and Optimal Inference. Translations of mathematical monographs, 53:477–493, 1982.
R. Collobert and J. Weston. A unified architecture for natural language processing: Deep
neural networks with multitask learning. In ICML’08, pages 160–167, 2008.
32

Y. Dauphin and Y. Bengio.
abs/1301.3583, 2013.

Big neural networks waste capacity.

arXiv CoRR,

G. Desjardins, A. C. Courville, and Y. Bengio. On training deep boltzmann machines.
CoRR, abs/1203.4416, 2012.
R. W. Duin and E. Peȩkalska. Object representation, sample size, and data set complexity.
In Data Complexity in Pattern Recognition, pages 25–58. Springer London, 2006.
D. Erhan, Y. Bengio, A. Courville, P-A. Manzagol, P. Vincent, and S. Bengio. Why does
unsupervised pre-training help deep learning? Journal of Machine Learning Research,
11:625–660, 2010.
I. Fodor. A survey of dimension reduction techniques. Technical report, Center for Applied
Scientic Computing, Lawrence Livermore National Laboratory, United States, 2002.
B. R. Frieden. Science from Fisher Information: A Unification. Cambridge University
Press, 2004.
W. R. Gilks, S. Richardson, and D.J. Spiegelhalter. Markov Chain Monte Carlo in Practice.
Chapman and Hall/CRC, 1996.
G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural
Comput., 14(8):1771–1800, 2002.
G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural
networks. Science, 313(5786):504–507, 2006.
Y. Hou, X. Zhao, D. Song, and W. Li. Mining pure high-order word associations via
information geometry for information retrieval. ACM TOIS, 31(3), 2013.
I. T. Jolliffe. Principal component analysis. Springer Series in Statistics, New York, US,
2002.
R. E. Kass. The geometry of asymptotic inference. Statistical Science, 4(3):188–219, 1989.
John A. Lee and Michel Verleysen, editors. Nonlinear Dimensionality Reduction. Springer,
New York, US, 2007.
H. Nakahara and S. Amari. Information geometric measure for neural spikes. Neural
Computation, 14:2269–2316, 2002.
S. Osindero and G. E. Hinton. Modeling image patches with a directed hierarchy of markov
random field. In NIPS’07, pages 1121–1128, 2007.
M. Ranzato, C. Poultney, S. Chopra, and Y. LeCun. Efficient learning of sparse representations with an energy-based model. In NIPS’06, pages 1137–1144, 2006.
C. R. Rao. Information and the accuracy attainable in the estimation of statistical parameters. Bulletin of calcutta mathematics society, 37:81–89, 1945.
33

S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio. Contractive auto-encoders:
Explicit invariance during feature extraction. In ICML’11, pages 833–840, 2011.
R. Salakhutdinov and G. E. Hinton. Using deep belief nets to learn covariance kernels for
gaussian processes. In NIPS’07, pages 1249–1256, 2007a.
R. Salakhutdinov and G. E. Hinton. Semantic hashing. In Workshop SIGIR’07, 2007b.
R. Salakhutdinov and G. E. Hinton. An efficient learning procedure for deep boltzmann
machines. Neural Computing, 24(8):1967–2006, 2012.
P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P-A. Manzagol. Stacked denoising
autoencoders: Learning useful representations in a deep network with a local denoising
criterion. J. Mach. Learn. Res., 11:3371–3408, 2010.
J. A. Wheeler. Time today. In Physical Origins of Time Asymmetry, pages 1–29. Cambridge
University Press, 1994.
L. Younes. On the convergence of markovian stochastic algorithms with rapidly decreasing
ergodicity rates. In Stochastics and Stochastics Models, pages 177–228, 1998.

34

