Algebraic geometrical methods for hierarchical learning machines.
Hierarchical learning machines such as layered perceptrons, radial basis functions, Gaussian mixtures are non-identifiable learning machines, whose Fisher information matrices are not positive definite. This fact shows that conventional statistical asymptotic theory cannot be applied to neural network learning theory, for example either the Bayesian a posteriori probability distribution does not converge to the Gaussian distribution, or the generalization error is not in proportion to the number of parameters. The purpose of this paper is to overcome this problem and to clarify the relation between the learning curve of a hierarchical learning machine and the algebraic geometrical structure of the parameter space. We establish an algorithm to calculate the Bayesian stochastic complexity based on blowing-up technology in algebraic geometry and prove that the Bayesian generalization error of a hierarchical learning machine is smaller than that of a regular statistical model, even if the true distribution is not contained in the parametric model.
Introduction
Learning in artificial neural networks can be understood as statistical estimation
of an unknown probability distribution based on empirical samples (White, 1989;
Watanabe & Fukumizu, 1995). Let p(y|x, w) be a conditional probability density
function which represents a probabilistic inference of an artificial neural network,
where x is an input and y is an output. The parameter w, which consists of a lot
of weights and biases, is optimized so that the inference p(y|x, w) approximates the
true conditional probability density from which training samples are taken.
Let us reconsider a basic property of a homogeneous and hierarchical learning
machine. If the mapping from a parameter w to the conditional probability density
p(y|x, w) is one-to-one, then the model is called identifiable. If otherwise, then it is
called non-identifiable. In other words, a model is identifiable if and only if its parameter
is uniquely determined from its behavior. The standard asymptotic theory
in mathematical statistics requires that a given model should be identifiable. For
example, identifiablity is a necessary condition to ensure that both the distribution
of the maximum likelihood estimator and the Bayesian a posteriori probability density
function converge to the normal distribution if the number of training samples
tends to infinity (Cramer, 1949). When we approximate the likelihood function
by a quadratic form of the parameter and select the optimal model using information
criteria such as AIC, BIC, and MDL, we implicitly assume that the model is
identifiable.
However, many kinds of artificial neural networks such as layered perceptrons, radial
basis functions, Boltzmann machines, and gaussian mixtures are non-identifiable,
hence either their statistical property is not yet clarified or conventional statistical
design methods can not be applied. In fact, a failure of likelihood asymptotics for
normal mixtures was shown from the viewpoint of testing hypothesis in statistics
(Hartigan, 1985). In researches of artificial neural networks, it was pointed out
that AIC does not correspond to the generalization error by the maximum likelihood
method (Hagiwara, 1993), since the Fisher information matrix is degenerate
if the parameter represents the smaller model (Fukumizu, 1996). The asymptotic
distribution of the maximum likelihood estimator of a non-identifiable model was
analyzed based on the theorem that the empirical likelihood function converges to
the gaussian process if it satisfies Donsker's condition (Dacunha-Castelle & Gassiat,
1997). It was proven that the generalization error by the Bayesian estimation is far
smaller than the number of parameters divided by the number of training samples
(Watanabe, 1997; Watanabe, 1998). When the parameter space is conic and sym-
metric, the generalization error of the maximum likelihood method is di#erent from
that of a regular statistical model (Amari & Ozeki, 2000). If the log likelihood function
is analytic for the parameter and if the set of parameters is compact, then the
generalization error by the maximum likelihood method is bounded by the constant
divided by the number of training samples (Watanabe, 2001b).
Let us illustrate the problem caused by non-identifiability of layered learning
machines. If p(y|x, w) be a three-layer perceptron with K hidden units and if w 0
is a parameter such that p(y|x, w 0 ) is equal to the machine with K 0 hidden units
then the set of true parameters
consists of several sub-manifolds in the parameter space. Moreover, the Fisher
information matrix,
log p(y|x, w)
log p(y|x, w)p(y|x, w)q(x)dxdy,
where q(x) is the probability density function on the input space, is positive semi-definite
but not positive definite, and its rank, rank I(w), depends on the parameter
This fact indicates that artificial neural networks have many singular points in
the parameter space (Figure 1). A typical example is shown in Example.2 in section
3. By the same reason, almost all homogenous and hierarchical learning machines
such as a Boltzmann machine, a gaussian mixture, and a competitive neural network
have singularities in their parameter spaces, resulting that we have no mathematical
foundation to analyze their learning.
In the previous paper (Watanabe, 1999b; Watanabe, 2000; Watanabe, 2001a), in
order to overcome such a problem, we proved the basic mathematical relation between
the algebraic geometrical structure of singularities in the parameter space and
the asymptotic behavior of the learning curve, and constructed a general formula to
calculate the asymptotic form of the Bayesian generalization error using resolution
of singularities, based on the assumption that the true distribution is contained in
the parametric model.
In this paper, we consider a three-layer perceptron in the case when the true
probability density is not contained in the parametric model, and clarify how singularities
in the parameter space a#ect learning in Bayesian estimation. By employing
an algebraic geometrical method, we show the following facts.
(1) The learning curve is strongly a#ected by singularities, since the statistical estimation
error depends on the estimated parameter.
(2) The learning e#ciency can be evaluated by using the blowing-up technology in
algebraic geometry.
(3) The generalization error is made smaller by singularities, if the Bayesian estimation
is applied.
These results clarify the reason why the Bayesian estimation is useful in practical
applications of neural networks, and demonstrate a possibility that algebraic geometry
plays an important role in learning theory of hierarchical learning machines,
just same as di#erential geometry did in that of regular statistical models (Amari,
1985).
This paper consists of 7 sections. In section 2, the general framework of Bayesian
estimation is formulated. In section 3, we analyze a parametric case when the
true probability density function is contained in the learning model, and derive the
asymptotic expansion of the stochastic complexity using resolution of singularities.
In section 4, we also study a non-parametric case when the true probability density
is not contained, and clarify the e#ect of singularities in the parameter space. In
section 5, the problem of the asymptotic expansion of the generalization error is
considered. Finally, section 6 and 7 are devoted to discussion and conclusion.
Bayesian Framework
In this section, we formulate the standard framework of Bayesian estimation and
Bayesian stochastic complexity (Schwarz 1974; Akaike, 1980; Levin, Tishby, & Solla,
1990; Mackay, 1992; Amari, Fujita, & Shinomoto, 1992; Amari & Murata, 1993).
Let p(y|x, w) be a probability density function of a learning machine, where an
input x, an output y, and a parameter w are M , N , and d dimensional vectors,
respectively. Let q(y|x)q(x) be a true probability density function on the input
and out space, from which training samples {(x i , y i are independently
taken. In this paper, we mainly consider the Bayesian framework, hence the
estimated probability density # n (w) on the parameter space is defined by
exp(-nH n (w))#(w),
log
where Z n is the normalizing constant, #(w) is an arbitrary fixed probability density
function on the parameter space called an a priori distribution, and H n (w) is the
empirical Kullback distance. Note that the a posteriori distribution # n (w) does
not depend on {q(y i |x i constant function of w.
Hence it can be written in the other form,
The inference p n (y|x) of the trained machine for a new input x is defined by the
average conditional probability density function,
The generalization error G(n) is defined by the Kullback distance of p n (y|x) from
q(y|x),
{ # q(y|x) log
q(x)dxdy}, (1)
represents the expectation value overall sets of training samples. One
of the most important purposes in learning theory is to clarify the behavior of the
generalization error when the number of training samples are su#ciently large.
It is well known (Levin, Tishby, Solla, 1990; Amari, 1993; Amari, Murata, 1993)
that the generalization error G(n) is equal to the increase of the stochastic complexity
F (n),
for an arbitrary natural number n, where F (n) is defined by
The stochastic complexity F (n) and its generalized concepts, which are sometimes
called the free energy, the Bayesian factor, or the logarithm of the evidence, can
be seen in statistics, information theory, learning theory, and mathematical physics
(Schwarz, 1974; Akaike, 1980; Rissanen, 1986; Mackay, 1992; Opper & Haussler,
1995; Meir & Merhav, 1995 ; Haussler & Opper, 1997; Yamanishi, 1998). For
example, both Bayesian model selection and hyperparatemeter optimization are
often carried out by minimization of the stochastic complexity before averaging.
They are called BIC and ABIC, which are important in practical applications.
The stochastic complexity satisfies two basic inequalities. Firstly, we define H(w)
and F (n) respectively by
q(x)dxdy,
Note that H(w) is called the Kullback information. Then, by applying Jensen's
inequality,
holds for an arbitrary natural number n (Opper & Haussler, 1995; Watanabe, 2001a).
Secondly, we use notations F (#, n) = F (n) and F (#, n) = F (n) which explicitly
show the a priori probability density #(w). Then F (#, n) and F (#, n) can be understood
as a generalized stochastic complexity for a case when #(w) is a non-negative
function. If #(w) and #(w) satisfy
then it immediately follows that
Therefore, the restriction of the integrated region of the parameter space makes the
stochastic complexity not smaller. For example, we define
exp(-nH(w))#(w)dw, (7)
with su#ciently small # > 0, then
These two inequalities eq.(4) and eq.(8) give upper bounds of the stochastic com-
plexity. On the other hand, if the support of #(w) is compact, then a lower bound
is proven
Moreover, if the learning machine contains the true distribution, then
holds (Watanabe, 1999b; Watanabe, 2001a).
In this paper, based on algebraic geometrical methods, we prove rigorously the
upper bounds of F (n) such as
are constants and o(log n) is a function of n which satisfies o(log n)/ log n #
Mathematically speaking, although the generalization error G(n) is
equal to F (n natural number n, we can not derive the asymptotic
expansion of G(n). However, in section 5, we show that, if G(n) has some
asymptotic expansion, then it should satisfy the inequality
for su#ciently large n, from eq.(11). The main results of this paper are the upper
bounds of the stochastic complexity, however, we also discuss the behavior of the
generalization errors based on eq.(12).
3 A Parametric Case
In this section, we consider a parametric case when the true probability distribution
q(y|x)q(x) is contained in the learning machine p(y|x, w)q(x), and show the relation
between the algebraic geometrical structure of the machine and the asymptotic form
of the stochastic complexity.
3.1 Algebraic Geometry of Neural Networks
In this subsection, we briefly summarize the essential result of the previous paper.
For the mathematical proofs of this subsection, see (Watanabe, 1999b; Watanabe,
2001a). Strictly speaking, we need assumptions that log p(y|x, w) is an analytic
function of w, and that it can be analytically continued to a holomorphic function
of w whose associated convergence radii is positive uniformly for arbitrary (x, y)
that satisfies q(y|x)q(x) > 0 (Watanabe, 2000; Watanabe, 2001a). In this paper, we
apply the result of the previous paper to the three-layer perceptron.
If a three-layer perceptron is redundant to approximate the true distribution,
then the set of true parameters {w; is a union of several sub-manifolds
in the parameter space. In general, the set of all zero points of an analytic function
is called an analytic set. If the analytic function H(w) is a polynomial, then the set
is called an algebraic variety. It is well known that an analytic set and an algebraic
variety have complicated singularities in general.
We introduce a state density function v(t)
where #(t) is Dirac's delta function and # > 0 is a su#ciently small constant. By
definition, if t < 0 or t > #, then using v(t), F # (n) is rewritten as
exp(-nH(w))#(w)dw
dt
.
Hence, if v(t) has an asymptotic expansion for t # 0, then F # (n) (n #) has an
asymptotic expansion for n #.
In order to examine v(t), we introduce a kind of the zeta function J(z) (Sato
of the Kullback information H(w) and the a priori probability
density #(w), which is a function of one complex variable z,
H(w) z #(w)dw (14)
Then J(z) is an analytic function of z in the region Re(z) > 0. It is well known in
the theory of distributions and hyperfunctions that, if H(w) is an analytic function
of w, then J(z) can be analytically continued to a meromorphic function on the
entire complex plane and its poles are on the negative part of the real axis (Atiyah,
1970; Bernstein, 1972; Sato & Shintani, 1974; Bj-ork, 1979). Moreover, the poles of
J(z) are rational numbers (Kashiwara, 1976). Let -# 1
be the largest pole and its order of J(z), respectively. Note that eq.(15) shows J(z)
(z # C) is the Mellin transform of v(t). Using the inverse Mellin transform, we can
show that v(t) satisfies
where c 0 > 0 is a positive constant. By eq.(13), F # (n) has an asymptotic expansion,
where O(1) is a bounded function of n. Hence, by eq.(8),
Moreover, if the support of #(w) is a compact set, by eq.(9), we obtain an asymptotic
expansion of F (n),
We have the first theorem.
Theorem 1 (Watanabe, 1999b; Watanabe, 2001a) Assume that the support of #(w)
is a compact set. The stochastic complexity F (n) has an asymptotic expansion,
are respectively the largest pole and its order of the function that
is analytically continued from
H(w) z #(w)dw,
where H(w) is the Kullback information and #(w) is the a priori probability density
function.
Remark that, if the support of #(w) is not compact, then Theorem 1 gives an upper
bound of F (n).
The important constants # 1 and m 1 can be calculated by an algebraic geometrical
method. We define the set of parameters W # by
It is proven by Hironaka's resolution theorem (Hironaka, 1964 ; Atiyah, 1970) that
there exist both a manifold U and a resolution map
d
in an arbitrary neighborhood of an arbitrary u # U that satisfies
where a(u) > 0 is a strictly positive function and {k i } are non-negative even integers

Figure

2). Let
be a decomposition of W # into a finite union of suitable neighborhoods W # , where
By applying the resolution theorem to the function J(z),
H(w) z #(w)dw
H(w) z #(w)dw
is given by recursive blowing-ups, the Jacobian |g # (u)|
is a direct product local variables u 1 ,
d
where c(u) is a positive analytic function and {h j } are non-negative integers. In a
neighborhood U # , a(u) and #(g(u)) can be set as constant functions in calculation
of the poles of J(z), because we can take each U # small enough. Hence we can set
loss of generality. Then,
d
where both k (#)
j depend on the neighborhood U # . We find that J(z) has
poles {-(h (#)
j }, which are rational numbers on the negative part of the
real axis.
Since a resolution map g(u) can be found by using finite recursive procedures of
blowing-ups, # 1 and m 1 can be found algorithmically. It is also proven that # 1 # d/2
if {w; #(w) > 0, #, and that m 1 # d.
Theorem 2 (Watanabe, 1999b; Watanabe, 2001a) The largest pole -# 1 and its
of the function J(z) can be algorithmically calculated by Hironaka's
resolution theorem. Moreover, # 1 is a rational number and m 1 is a natural number,
and if {w; #(w) > 0,
where d is the dimension of parameter.
Note that, if the learning machine is a regular statistical model, then always #
Also note that, if Je#reys' prior is employed in neural network learning,
which is equal to zero at singularities, the assumption {w; #(w) > 0,
is not satisfied, and then both # even if the Fisher metric
is degenerate (Watanabe, 2001c).
Example.1 (Regular Model) Let us consider a regular statistical model
exp(-2
with the set of parameters Assume that the true
distribution is
exp(-2
and the a priori distribution is the uniform distribution on W . Then,
For a subset S # W , we define
Then
We introduce a mapping
Then
=2 z
has a pole at z = -1. We can show JW 2
(z) has the same pole just the same way as
. Hence # resulting in F (n) log n. This coincides with the
well known result of the Bayesian asymptotic theory of regular statistical models.
The mapping in eq.(17) is a typical example of a blowing-up.
Example.2 (Non-identifiable model) Let us consider a learning machine,
p(y|x, a, b, c) =# 2#
exp(-2
Assume that the true distribution is same as eq.(16), and that
the a priori probability distribution is the uniform one on the set
Then, the Kullback information is
Let us define two sets of parameters,
|
By using blowing-ups recursively, we find a map which is defined by
By using this transform, we obtain
Therefore,
H(w) z dw
=2 z
The largest pole of JW 1 (z) is -3/4 and its order is one. It is also shown that JW\W 1 (z)
have largest pole -3/4 with order one. Hence # resulting that
log n +O(1).
3.2 Application to Layered Perceptron
We apply the theory in the foregoing subsection to the three-layer perceptron. A
three-layer perceptron with the parameter defined by
a k #(b k - x
where y, f(x, w), and a h are N dimensional vectors, x and b h are M dimensional
vectors, c h is a real number, and and K are the
numbers of input units, output units, and hidden units. In this paper, we consider
a machine which does not estimate the standard deviation s > 0 (s is a constant).
We assume that the true distribution is
That is to say, the true regression function is This is a special case, but
analysis of this case is important in the following section where the true regression
function is not contained in the model.
Theorem 3 Assume that the learning machine given by eq.(18) and eq.(19) is
trained using samples independently taken from the distribution, eq.(20). If the a
priori distribution satisfies #(w) > 0 in the neighborhood of the origin
(Proof of Theorem We use notations,
a
Then the Kullback information is
(b, c) a hp a kp ,
where
Our purpose is to find the pole of the function
where
Let us apply the blowing-up technique to the Kullback information H(a, b, c).
Firstly, we introduce a mapping
which is defined by
a
a
Let u # be the variables of u except u 11 , in other words,
where
and the Jacobian |g # (u)| of the mapping g is
We define a set of paramaters for # > 0
By the assumption, there exists # > 0 such that
In order to obtain an upper bound of the stochastic complexity, we can restrict the
integrated region of the parameter space, by using eq.(5) and (6).
By the assumption #(w) > 0 in g(U(#)). In calculation of the pole of J(z), we can
assume is a constant) in g(U(#)).
du # db dc
The pole of the function #
respectively the largest poles of J(z) and
Then, since H 1 does not have zero point in the interval (-# 1 , #).
larger than -# 1 , then z = -NK/2 is a pole of J(z). If otherwise,
then J(z) has a larger pole than -NK/2. Hence # 1 # NK/2.
Secondly, we consider another blowing-up g,
which is defined by
Then, just the same method as the first half, there exists an analytic function
which implies
Therefore
By combing the above two results, the largest pole -# 1 of the J(z) satisfies the
inequality,
which completes the proof of Theorem 3. (End of Proof).
By Theorem 1,
Moreover, if G(n) has an asymptotic expansion (see section 5), we obtain an inequality
of the generalization error,
On the other hand, it is well known that the largest pole of a regular statistical
model is equal to -d/2, where d is the number of parameters. When a three-layer
perceptorn with 100 input units, 10 hidden units, and 1 output unit is employed, then
the regular statistical models with the same number of parameters
has It should be emphasized that the generalization error of
the hierarchical learning machine is far smaller than that of the regular statistical
models, if we use the Bayesian estimation.
When we adopt the normal distribution as the a priori probability density, we
have shown the same result as Theorem 3 by a direct calculation (Watanabe, 1999a).
However, Theorem 3 shows systematically that the same result holds for an arbitrary
a priori distribution. Moreover, it is easy to generalize the above result to the case
when the learning machine has M input units, K 1 first hidden units, K 2 second
hidden units, ., K p pth hidden units, and N output units. We assume that hidden
units and output units have bias parameters. Then by using same blowing-ups, we
can generalize the proof of Theorem 3,
Of course, this result holds only when the true regression function is the special
case, However, in the following section, we show that this result is necessary
to obtain a bound for a general regression function.
4 A Non-parametric Case
In the previous section, we have studied a case when the true probability distribution
is contained in the parametric model. In this section, we consider a non-parametric
case when the true distribution is not contained in the parametric models, which is
illustrated in Figure 3.
Let w 0 be the parameter that minimizes H(w), which is a point C in Figure
3. Our main purpose is to clarify the e#ect of singular points such as A and B in

Figure

3 which are not contained in the neighborhood of w 0 . Let us consider a case
when a three-layer perceptron given by eq.(18) and eq.(19) is trained using samples
independently taken from the true probability distribution,
where g(x) is the true regression function and q(x) is the true probability distribution
on the input space. Let E(k) be the minimum function approximation error using
a three-layer perceptron with k hidden units,
Here we assume that, for each 1 # k # K, there exists a parameter w that attains
the minimum value.
Theorem 4 Assume that the learning machine given by eq.(18) and eq.(19) is
trained using samples independently taken from the distribution of eq.(21). If the a
priori distribution satisfies #(w) > 0 for an arbitrary w, then
{
(D
where
(Proof of Theorem 4) By Jensen's inequality eq.(4), we have
where H(w) is the Kullback distance,
be natural numbers which satisfy both 0 # k 1 # K and
We divide the parameter
Also let # 1 and # 2 be real numbers which satisfy both # 1 > 1 and
Then, for arbitrary u, v # R N ,
Therefore, for arbitrary (x, w),
Hence we have an inequality,
where we use definitions,
As F (n) is an increasing function of H(w),
where
are some functions which satisfy
Here we can choose both # 1 (w 1 ) and # 2 (w 2 ) which are compact support functions.
Firstly, we evaluate F 1 (n). Let w # 1 be the parameter that minimizes H 1 (w 1 ).
Then, by eq.(22) and Theorem 2,
is the number of parameters in the three-layer perceptron
with k 1 hidden units.
Secondly, by applying Theorem 3 to F 2 (n),
By combining eq.(23) with eq.(24), and by taking # 1 su#ciently close 1, we obtain
{
for an arbitrary given
we Theorem 4. (End of Proof).
Based on Theorem 4, if G(n) has an asymptotic expansion (see section 5), then G(n)
should satisfy the inequalities
for n > n 0 with a su#ciently large n 0 . Hence
{ E(k)
for n > n 0 with a su#ciently large n 0 . Figure 4 illustrates several learning curves
corresponding to k (0 # k # K). The generalization error G(n) is smaller than
every curve.
It is well known (Barron, 1994; Murata, 1996) that, if g(x) belongs to some kind
of function space, then
for su#ciently large k, where C(g) is a positive constant determined by the true
regression function g(x). Then,
{
ASYMPTOTIC PROPERTY OF THE GENERALIZATION ERROR 22
If both n and K are su#ciently large, and if
then, by choosing
The inequality (27) holds if n is su#ciently large. If n is su#ciently large but not
extensively large, then G(n) is bounded by the generalization error of the middle
size model. If n becomes larger, then it is bounded by that of the larger model, and
if n is extensively large, then it is bounded by that of the largest model. A complex
hierarchical learning machine contains a lot of smaller models in its own parameter
space as analytic sets with singularities, and chooses the appropriate model adaptively
for the number of training samples, if Bayesian estimation is applied. Such a
property is caused by the fact that the model is non-identifiable, and its quantitative
e#ect can be evaluated by using algebraic geometry.
5 Asymptotic Property of the Generalization Er-
ror
In this section, let us consider the asymptotic expansion of the generalization error.
By eq.(2), F (n) is equal to the accumulate generalization error,
where G(0) is defined by F (1). Hence, if G(n) has an asymptotic expansion for
#, then F (n) also has the asymptotic expansion. However, even if F (n)
has an asymptotic expansion, G(n) may not have an asymptotic expansion. In the
foregoing sections, we have proved that F (n) satisfies inequalities such as
are constants determined by the singularities and the true distribu-
tion. In order to mathematically derive an inequality of G(n) from eq.(30), we need
an assumption.
ASYMPTOTIC PROPERTY OF THE GENERALIZATION ERROR 23
Assumption (A) Assume that the generalization error G(n) has an asymptotic
expansion
a q s q (n)
where {a q } are real constants, s q (n) > 0 is a positive and non-increasing function
of n which satisfies
Based on this assumption, we have the following lemma.
Lemma 1 If G(n) satisfies the assumption (A) and if eq.(30) holds, then G(n)
satisfies an inequality,
(Proof) By the assumption (A)
which shows a 1 #. If a 1 < #, then eq.(35) holds. If a
ks 2 (k). By eq.(32),eq.(33), and eq.(34), t(k) # or t(k) # C (C > 0).
If t(k) #, then, for arbitrary M > 0, there exists k 0 such that
Hence
which contradicts eq.(36). Hence t(n) # C and a 2 C #. (End of Proof Lemma 1).
In this paper, we have proven the inequalities same as eq.(30) in Theorem 1, 2, 3,
and 4 without assumption (A). Then, we obtain corresponding inequalities same as
if we adopt the assumption (A). In other words, if G(n) has an asymptotic
expansion and if eq.(30) holds, then G(n) should satisfy eq.(35). It is conjectured
that natural learning machines satisfy the assumption (A). A su#cient condition for
the assumption (A) is that F (n) has an asymptotic expansion
R
a
1). For
example, if the learner is
p(y|x, a) =# 2#
exp(-2
where the a priori distribution of a is the standard normal distribution, and if the
true distribution is
}),
then, it is shown by direct calculation that the stochastic complexity has an asymptotic
expansion
Hence G(n) has an asymptotic expansion
c 2+2n
It is expected that, in a general case, G(n) has the same asymptotic expansion as
Assumption (A), however, mathematically speaking, the necessary and su#cient
condition for it is not yet established. This is an important problem in statistics
and learning theory for the future.
6 Discussion
In this section, universal phenomena which can be observed in hierarchical learning
machines.
6.1 Bias and variance at singularities
We consider a covering neighborhood of the parameter space,
where {W (w j )} are the su#ciently small neighborhood of the parameter w j which
The number J in eq.(38) is finite when compact. Then, the upper-bound
of the stochastic complexity can be rewritten as
exp(-H(w))#(w)dw
is the function approximation error of the parameter w j
H(w),
and V (w j ) is the statistical estimation error of the neighborhood of w j ,
(- log n) m(w j )-1
where c 0 > 0 is a constant. The values -#(w j ) and m(w j ) are respectively the
largest pole and its multiplicity of the meromorphic function
Note that B(w j ) and V (w j ) are called the bias and the variance, respectively. In
the Bayesian estimation, the neighborhood of the parameter w j that minimizes
is selected with the largest probability. In regular statistical models, the variance
does not depend on the parameter, in other words, #(w j
for an arbitrary parameter w j , hence the parameter that minimizes the function approximation
error is selected. On the other hand, in hierarchical learning machines,
the variance V (w j ) strongly depends on the parameter w j , and the parameter that
minimizes the sum of the bias and variance is selected. If the number of training
samples is large but not extensively large, parameters among the singular point A
in

Figure

3 that represents a middle size model, is automatically selected, resulting
in the smaller generalization error. As n increases, the larger but not largest model
B is selected. At last, if n becomes extensively large, then the parameter C that
minimizes the bias is selected. This is a universal phenomenon of hierarchical learning
machines, which indicates the essential di#erence between the regular statistical
models and artificial neural networks.
6.2 Neural networks are over-complete basis
Singularities of a hierarchical learning machine originate in the homogeneous structure
of a learning model. A set of functions used in an artificial neural network, for
example, is a set of over-complete basis, in other words, coe#cients
{a(b, c)} in a wavelet type decomposition of a given function g(x),
are not uniquely determined for g(x) (Chui, 1989; Murata, 1996). In practical
applications, the true probability distribution is seldom contained in a parametric
model, however, we adopt a model which almost approximates the true distribution
compared with the fluctuation caused by random samples,
a k #(b k - x
If we have an appropriate number of samples and choose an appropriate learning
model, it is expected that the model is in an almost redundant state, where output
functions of hidden units are almost linearly dependent. We expect that this paper
will be a mathematical foundation to study learning machines in such states.
7 Conclusion
We considered the case when the true distribution is not contained in the parametric
models made of hierarchical learning machines, and showed that the parameters
among singular points are selected by the Bayesian distribution, resulting in the
small generalization error. The quantitative e#ect of the singularities was clarified
based on the resolution of singularities in algebraic geometry. Even if the true
distribution is not contained in the parametric models, singularities strongly a#ect
and improve the learning curves. This is a universal phenomenon of the hierarchical
learning machines, which can be observed in almost all artificial neural networks.



--R

Likelihood and Bayes procedure.

A universal theorem on learning curves.
Four Types of Learning Curves.
Neural Computation
Statistical theory of learning curves under entropic loss.

Resolution of Singularities and Division of Distributions.
Communications of Pure and Applied Mathematics
Approximation and estimation bounds for artificial neural networks.
The analytic continuation of generalized functions with respect to a parameter.
Mathematical methods of statistics
An introduction to Wavelets.
Testing in locally conic models

Generalized functions.
On the problem of applying AIC to determine the structure of a layered feed-forward neural network
A Failure of likelihood asymptotics for normal mixtures.

Mutual information
Resolution of singularities of an algebraic variety over a field of characteristic zero.

A statistical approaches to learning and generalization in layered neural networks.
Bayesian interpolation.
On the stochastic complexity of learning realizable and unrealizable rules.
An integral representation with ridge functions and approximation bounds of three-layered network
Bounds for predictive errors in the statistical mechanics of supervised learning.
Stochastic complexity and modeling.
On zeta functions associated with prehomogeneous vector space.

A optimization method of layered neural networks based on the modified information criterion.
On the essential di
On the generalization error by a layered statistical model with Bayesian estimation.


Algebraic analysis for non-regular learning machines

Neural Computation


Probabilistic design of layered neural networks based on their unified framework.
Learning in artificial neural networks: a statistical prespective.
Neural Computation
A decision-theoretic extension of stochastic complexity and its applications to learning
--TR
Bayesian interpolation
Four types of learning curves
A universal theorem on learning curves
An introduction to wavelets
Statistical theory of learning curves under entropic loss criterion
Approximation and Estimation Bounds for Artificial Neural Networks
On the Stochastic Complexity of Learning Realizable and Unrealizable Rules
A regularity condition of the information matrix of a multilayer perceptron network
An integral representation of functions using three-layered networks and their approximation bounds
Algebraic Analysis for Singular Statistical Estimation

--CTR
Miki Aoyagi , Sumio Watanabe, Stochastic complexities of reduced rank regression in Bayesian estimation, Neural Networks, v.18 n.7, p.924-933, September 2005
Keisuke Yamazaki , Sumio Watanabe, Singularities in mixture models and upper bounds of stochastic complexity, Neural Networks, v.16 n.7, p.1029-1038, September
Sumio Watanabe , Shun-ichi Amari, Learning coefficients of layered models when the true distribution mismatches the singularities, Neural Computation, v.15 n.5, p.1013-1033, May
Shun-Ichi Amari , Hiroyuki Nakahara, Difficulty of Singularity in Population Coding, Neural Computation, v.17 n.4, p.839-858, April 2005
Haikun Wei , Jun Zhang , Florent Cousseau , Tomoko Ozeki , Shun-ichi Amari, Dynamics of learning near singularities in layered networks, Neural Computation, v.20 n.3, p.813-843, March 2008
Shun-Ichi Amari , Hyeyoung Park , Tomoko Ozeki, Singularities Affect Dynamics of Learning in Neuromanifolds, Neural Computation, v.18 n.5, p.1007-1065, May 2006
resolution of singularities;generalization error;stochastic complexity;asymptotic expansion;algebraic geometry;non-identifiable model