A Common Framework for Natural Gradient and Taylor based Optimisation using Manifold Theory
Adnan Haider



A quasar hiding behind two dusty absorbers
    
K. E. Heintz1,2,
J. P. U. Fynbo2,
C. Ledoux3,
P. Jakobsson1,
P. Møller4,
L. Christensen2,
S. Geier5,6, 

J.-K. Krogager7
P. Noterdaeme7

    Received 2017; accepted, 2017
=============================================================================================================================================




This  technical report introduces a mathematical framework  to compare and relate  standard Taylor  approximation based optimisation methods with the method of Natural Gradient, an optimisation approach that is Fisher efficient with probabilistic models.  The constructed framework will be shown to also provide  a mathematical justification to combine  higher order Taylor approximation methods with the method of NG.  





§ INTRODUCTION



The core premise behind first and second order optimisation methods is Taylor's theorem.  Assuming that  the objective function F(θ) is sufficiently smooth, Taylor's second order approximation  models the local behaviour of the  function  by the following quadratic function: 


    F(θ_k + Δθ) ≃ F(θ_k ) +  Δθ^T  ∇ F(θ_k )+ 1/2Δθ^T H  Δθ

where Δθ represents any offset  within a convex neighbourhood of   θ_k and H is the Hessian of F computed w.r.t θ_k. Instead of optimising the objective function directly, second order methods focus on minimising the above approximate quadratic  at each iteration of the optimisation process. The same approach is undertaken by first order methods but  such approaches make a further  approximation by dropping the quadratic term in Taylor's formula.  This technical report provides a re-derivation of Taylor's theorem from the perspective of manifold theory.   Such a formulation will be shown  to provide a consistent framework to compare the above  approaches with the method of Natural Gradient <cit.> for  probabilistic discriminative models P_θ(ℋ|𝒪).  The necessary machinery needed to develop this framework relies on the concept of  manifold, tangent vectors and directional derivatives from the perspective of Information Geometry. Appendix <ref> contains a glossary of terms referenced in this work and   a more in-depth discussion can be found in Amari's text book <cit.>. 














§ DERIVING TAYLOR'S THEOREM USING MANIFOLD THEORY 



A   curve on a parameter manifold (Appx. <ref>) X is a continuous  map  c : (a, b)  ⊂ℝ→  X.   Let 𝒰 be an open convex neighbourhood  of the current iterate θ_k. Thus for any point θ in  𝒰, ∃  a curve of the form θ_k + t (θ-θ_k) where t ∈ [0,1] that is contained in 𝒰.



 

Let c : [0, 1] →  X  be a  continuous curve such that:

    c(t) = θ_k + t Δθ

where  Δθ corresponds to arbitrary offset from θ_k such that c in contained in 𝒰.  The derivative of  c at given point t_0 is then the linear map from the tangent space (Appx. <ref>) at t_0 to the tangent space T_c(t_0)  X:

 
    d c  |_t_0 (   ddr )    =  ( ∂∂θ̂_1, ∂∂θ̂_2⋯∂∂θ̂_D) [
    [ Δθ_1; Δθ_2;    ⋮; Δθ_D ]] 
       =  ∑_i Δθ_i  ∂∂θ_i

 
 where ddr denotes   the  basis vector  of T_t_0ℝ.

 

 Let F(θ) be a germ (Appx. <ref>) that corresponds to  a smooth map from the parameter manifold X to ℝ. In the context of optimisation, this corresponds to the smooth objective training criterion.
 The derivative of  F(θ) at any given point θ is a  linear map  from the tangent space from  T_θ  X to the tangent space  T_F(θ)ℝ:
 
 
    d F  |_θ  ( ∑_i a^i∂/∂θ_i |_θ)    =  (   ddr )  [
    ( ∂ F ∂θ̂_1 |_θ , ∂ F ∂θ̂_2|_θ⋯∂ F∂θ̂_D|_θ )
    ] 
     [
    [ a^1; a^2;   ⋮; a^D;     ] ]

 Here the vector ∇ F(θ_k )  is the Jacobian and denotes the particular vector in  T_θ M that yields the greatest directional derivative (Appx. <ref>). 




Constraining F on the curve c is equivalent to applying  the composite map  F ∘ c  from ℝ→ℝ. The derivative of such a map   at given point t_0 will now correspond to a linear map from a tangent space in T_t_0ℝ to the tangent space of  T_F∘ c(t_0)ℝ.  By applying chain rule such a linear map can be shown to correspond to:

    d (F  ∘  c ) |_t_0 (   ddr )     =  (ddr)  [ F  ∘  c ]_t_0
       =  (   ddr)   [
    ( ∂ F ∂θ̂_1 |_θ_k + t_0 Δθ , ∂ F ∂θ̂_2|_θ_k + t_0 Δθ⋯∂ F∂θ̂_D|_θ_k + t_0 Δθ )
    ] [
    [ Δθ_1; Δθ_2;    ⋮; Δθ_D ]]  
       =  ddr∑_i Δθ_i  ∂ F∂θ_i |_θ_k + t Δθ
 






Using the fact that F ∘ c is differentiable and corresponds to a map from  ℝ→ℝ,  by the  fundamental theorem of calculus: 


    F(θ_k + Δθ)    =  F(θ_k )+  _0^1  ( F∘ c(t)  )' dt 
       =F(θ_k ) + _0^1∑_i Δθ_i  ∂  F/∂θ_i (θ_k + t Δθ) dt  
       =F(θ_k ) + ∑_i Δθ_i   _0^1∂  F/∂θ_i (θ_k + t Δθ) dt


Since   individual terms ( _0^1∂  F∂θ_i ( θ_k + t Δθ dt )  themselves are  smooth functions defined on the convex neighbourhood of θ_k, (<ref>) can be expanded  even further by recursively applying the  fundamental theorem of calculus:

    F(θ_k + Δθ)    =  F(θ_k ) + ∑_i ∂ F/∂θ_i(θ_k) Δθ_i   + ∑_i Δθ_i  _0^1( ∑_j Δθ_j  _0^1∂ ^2 F/∂θ_j∂θ_i(θ_k + t Δθ ) dt  ) dt 
       =F(θ_k ) + ∑_i ∂ F/∂θ_i(θ_k) Δθ_i   + ∑_i Δθ_i   ∑_j Δθ_j  _0^1( _0^1∂ ^2 F/∂θ_j∂θ_i(θ_k + t Δθ ) dt  ) dt




As the function ∂ ^2 F∂θ_j∂θ_i(θ) is continuous, when Δθ is sufficiently small,  ∂ ^2 F∂θ_j∂θ_i(θ_k + t Δθ) can be approximated by ∂ ^2 F∂θ_j∂θ_i(θ_k + t Δθ ) |_t =0 .  Under this  approximation, the local behaviour of the germ F(θ) can be approximated by:





    F(θ_k + Δθ)    ≃  F(θ_k ) + ∑_i ∂ F/∂θ_i (θ_k) Δθ_i   + ∑_i Δθ_i  _0^1( ∑_j Δθ_j  _0^1∂ ^2 F/∂θ_j∂θ_i(θ_k ) dt  ) dt 
       ≃  F(θ_k ) + ∑_i ∂ F/∂θ_i (θ_k)Δθ_i   +  1/2∑_i Δθ_i  ∑_j Δθ_j  ∂ ^2 F/∂θ_j∂θ_i(θ_k)



In vector notation this corresponds to:

    F(θ_k + Δθ) ≃  F(θ_k ) +  Δθ^T  ∇ F(θ_k )+ 1/2Δθ^T H  Δθ

with entries  H_j,i of  the Hessian matrix corresponding to (  ∂ ^2 F∂θ_j∂θ_i(θ_k)  ).   The above expression corresponds to Taylor's second order approximation. Re-deriving this expression from the perspective of manifold theory shows how the  product Δθ^T  ∇ F(θ_k ) can be interpreted as an inner product between vectors Δθ and ∇ F(θ_k) in T_θ X. This interpretation will become handy in the next section. 



§  FORMULATING TAYLOR'S QUADRATIC AS A  MINIMISATION PROBLEM IN THE TANGENT SPACE 



Instead of   minimising the objective function directly, second order methods   focus on minimising the  quadratic function of (<ref>) at each iteration. The quadratic corresponds to a local model of the behaviour of F(θ) within a convex neighbourhood of the current iterate θ_k. By re-deriving Taylor's second order approximation from the perspective of manifold theory in the previous section, it was shown how Δθ corresponds to a particular choice of  a tangent vector from T_θ_k X, and  ∇ F(θ_k ) represents the particular vector in  T_θ X that yields the greatest directional derivative under the linear map d F  |_θ . Thus,  as  F(θ_k ) is a  constant,  solving the minimisation problem in (<ref>) is equivalent to solving the following minimisation problem in  T_θ_k X:


    Δθ̂ =  min_Δθ∈ T_θ_k X  F(θ_k ) +  ⟨Δθ,  ∇ F(θ_k ) ⟩ + 1/2Δθ^T H  Δθ

where ⟨Δθ,  ∇ F(θ_k ) ⟩  corresponds to the standard inner product between vectors in T_θ_k X   and  Δθ^T H  Δθ corresponds  to a linear map g : u∈  T_θ_k X →ℝ. 



 §.§ Relating Gradient Descent with Natural Gradient 



Under this framework, first order methods can be seen to solve the following optimisation problem in the  
tangent space T_θ_k X:

    Δθ̂ =  min_Δθ∈ T_θ_k X  F(θ_k ) +  ⟨Δθ,  ∇ F(θ_k ) ⟩


 Since X is a manifold,  the inner product endowed on the tangent space  T_θ X at any point θ need not be just the identity matrix. The parameter manifold X can be equipped with any form of a  Riemannian metric,  a smooth map that assigns to each θ∈ X an inner product  I_θ  in T_θ X.  When the underlying model  corresponds to a discriminative probabilistic model P_θ(ℋ|𝒪), a particular choice  of  I_θ is the Fisher Information matrix <cit.>:


    I_θ   =  E_ P_θ(ℋ|𝒪)  [ ( ∇ P_θ(ℋ|𝒪)  ) ( ∇ P_θ(ℋ|𝒪)  )^T  ]

Equipping the tangent space T_θ X with the above Riemannian metric  allows interpretation of  lengths of paths traversed in the parameter space as changes in the KL divergence (Appx. <ref> ). When X  possesses such a structure, performing  first order optimisation within a trust region defined by I_θ can be shown to produce the  update Δθ =I_θ^-1∇ F(θ ) at each iteration <cit.>.  From the perspective of  Information Geometry, this corresponds to the direction of steepest descent in the space of P_θ(ℋ|𝒪).  Thus, in this sense,  such an update is termed as the direction of Natural  Gradient (NG) <cit.> .

Using the fact that  I_θ  is symmetric and positive definite, it is also possible  to endow X with a Riemannian metric of the form   I_θ^-1  by the spectral decomposition theorem. With respect to such a metric,  solving the minimising problem of (<ref>) in the tangent space T_θ_k X  becomes equivalent to performing NG on the parameter surface. Hence, recasting the optimisation problem to a minimisation problem  in T_θ_k X provides a nice framework to relate  Gradient Decent with the method of NG. 

In practice, since it is not feasible to compute the expected  outer product of the likelihood score (<ref>) exactly,  I_θ  is approximated  by its Monte-Carlo estimate Î_θ.  As such a matrix  is only positive semi-definite, it's inverse is not guaranteed to exist. To address this issue,  this work also provides the derivation of  an  alternative dampened Riemannian metric Ĩ_θ^-1 (Appx. <ref>) that  is not only guaranteed to be positive definite but has the feature that its image space is the direct sum of the image  and  the kernel space of the empirical Fisher matrix Î_θ^-1. Assigning such a metric has one particular advantage: the very first directions explored by the CG algorithm constitute to directions in the image space of Î_θ <cit.> i.e  directions considered important by the empirical Fisher are  traversed first during the initial stages of a CG run.  




§ SUMMARY

To summarise, this report  presents a  mathematical framework  to compare and relate  standard Taylor  approximation based optimisation methods with the method of Natural Gradient, an optimisation approach that is Fisher efficient with probabilistic models. The constructed framework can be seen to also provide  a mathematical justification to combine  higher order Taylor approximation methods with the method of NG.  



 
 
 
 








§ GLOSSARY 




This section begins with  a formal description of the concept of a smooth manifold. Conceptually, a manifold corresponds to any geometric object embedded in ℝ^k that is locally Euclidean i.e any local patch of the object is  topologically equivalent to an open unit ball in a smaller dimensional  Euclidean space. In ℝ^3, curves and surfaces are  examples of  an embedded manifold. When movement is  constrained  only along  these objects, the degrees of freedom with which one can  traverse  is lower than dimensionality of the embedded 3 dimensional space. 




 §.§  Formal definition of  a manifold 

A topological manifold X of dimension D is a second countable Hausdorff topological space that is locally homeomorphic to ℝ^D;  that is, for any point θ∈ X, there exists an open neighbourhood U of  θ and a homeomorphism g : U → O ⊂ℝ^D, where O is open in ℝ^D. We call the homeomorphism g : U → O a chart, and the neighbourhood U a coordinate neighbourhood of θ.  



To clarify the reader, in topology the concept  of  a homeomorphism  is equivalent  to a bijective continuous map and  the concept of Hausdorff means for any two points θ̂,θ̅∈ X, it is always possible to find disjoint open neighbourhoods that contain each point.



  Let r_i: ℝ^D →ℝ denote the projection onto the ith coordinate. Given a chart g : U → O ⊂ℝ^D, let θ_i : r_i ∘ g : U →ℝ. The functions θ_i are  the local coordinates w.r.t the chart g on U.  Having established a notion of what it means to be manifold, 
the next concept that will be introduced is the concept of a tangent vector. To formally define the concept of a tangent vector at any point θ∈ X, it is first necessary to formalise the notion of germs associated with a given point in  a manifold.




 §.§ Germs 

Let X be a manifold and θ∈ X. Functions f, g defined on open subsets U, V respectively containing θ are said to have the same germ at θ  if there exists a neighbourhood W of θ contained in U ∩ V such that f |_W≡ g |_W.  The notion of germs 
therefore defines an equivalence relation on the space of functions defined on an open neighbourhood of  θ where  (U, f) ∼ (V, g) if and only if there exists
a neighbourhood W of θ contained in U ∩ V such that f |_W≡ g |_W.  Let C^∞_θ be the set of all such equivalent function classes.  Having defined the concept of class of germs associated  with a given point θ∈ X, the necessary machinery is now in place to define the concept of a tangent vector associated with the point θ∈ X.
  



 §.§ Tangent vector

A tangent vector v at  given point θ∈ X  is a linear derivation of  C^∞_θ, that is it a special form of  a linear map C^∞_θ→ℝ  that satisfies the property v(f  g) = f(θ)v(g) + v(f)g(θ) where f  g denotes the product of functions f  g  ∈ C^∞_θ. The tangent vectors form a real vector space in the obvious way;  this space is denoted by T_θ(X) and is called the tangent space to X at θ. 




The concept of tangent vectors is necessary  to do calculus on manifolds. Since manifolds are locally Euclidean, the usual notions of differentiation and integration make sense in any coordinate chart and  can be carried over to manifold space. Specifically, a tangent vector as will be shown now is the manifold version of a directional derivative at a point θ∈ X.

 Basis for Tangent space 
Let (θ_i,θ_2,⋯θ_D)    denote the standard coordinates of the parameter space X. Consider the operator ∂/∂θ_i |_θ defined   by ∂∂θ_i |_θ (f) = ∂ f∂θ_i(θ). Then the  set {∂/∂θ_i |_θ}_i can be shown to be  linear derivations of  C^∞_θ and hence members of  T_θ(X). Furthermore,  for ∀θ_j, each  operator in this set satisfies  

    ∂/∂θ_i |_θ(θ_j)  =
     
     1       i ≡ j   
    
     0       otherwise

It can be shown that, by satisfying the above constraint,   the members of {∂/∂θ_i |_θ}_i correspond to a basis of  T_θ(X).  Therefore, w.r.t this basis  if  v∈ T_θ(X) then  v = ∑_i a^i∂/∂θ_i |_θ.



 §.§ Concept of directional derivative

Let Φ: X → N  be a  vector valued function that corresponds to a smooth map between two manifolds. The directional derivative of Φ at  any point  θ∈ X is  the linear map 

    dΦ |_θ : T_θ(X) → T_Φ(θ)(N)

defined by:

    dΦ |_θ( ∑_i a^i∂/∂θ_i |_θ) =  (∂/∂ y_i |_Φ (θ),∂/∂ y_i |_Φ (θ),⋯∂/∂ y_k |_Φ (θ) ) [
    [    ∂Φ_1/∂θ_1 |_θ ∂Φ_1/∂θ_2 |_θ_2⋯;                 ;    ∂Φ_2/∂θ_1 |_θ ∂Φ_2/∂θ_2 |_θ_2⋯;                ⋮;    ∂Φ_k/∂θ_1 |_θ ∂Φ_k/∂θ_2 |_θ_2⋯ ]] 
     [
    [ a^1; a^2;   ⋮; a^D;     ] ]

where   {∂/∂ y_j |_Φ(θ)}_j  denotes the basis of T_Φ(θ)(N). Each coordinate of ∂/∂ y_j |_Φ(θ) corresponds to the directional derivative of Φ_j w.r.t θ. 






§ 



 §.§ Approximating the KL divergence using the Fisher Information Matrix 



Let X  denote  the  parameter manifold.  As different realisations of model parameters lead to different probabilistic models  P_θ(ℋ|𝒪), the manifold  is homeomorphic or in other words equivalent to the space of all probability distributions ℳ that can be captured by the chosen model. The goal of learning is to identify  a viable candidate h(θ|𝒪) ∈ℳ that  avoids rote memorisation and instead generalises to  the concepts that can be learned from a given set of examples. 

Using the fact that each candidate  h(θ|𝒪)  is a valid probability distribution, the derivatives of such densities satisfy the identity: 

    ∑_H  ∂/∂θ^ih(θ|𝒪)  =  ∑_H  ∂/∂θ_iP_θ(ℋ|𝒪)  =∂/∂θ_i∑_H P_θ(ℋ|𝒪)  =   ∂/∂θ_i  1 =0


Since  X is homeomorphic to ℳ, adding a small quantity Δθ to the current iterate θ  results in a unique distribution 
P_θ+ Δθ(ℋ|𝒪). To quantify the degree of separation between this point from the previous point  P_θ(ℋ|𝒪) in ℳ,  a standard  divergence measure for probabilistic models is  the KL-divergence KL  (P_θ(ℋ,|𝒪) P_θ+Δθ(ℋ|𝒪)).  




 The KL-divergence is  a functional that maps the space of  distributions  ℳ to ℝ.  Since each distribution itself is a function of θ, the divergence measure can be interpreted as a smooth function of θ. This allows the local  behaviour of this divergence measure to be approximated within a convex neighbourhood of the current point θ by Taylor's second order approximation:
 

    KL  (P_θ(ℋ|𝒪) P_θ+Δθ(ℋ|𝒪))  =   E_P_θ(ℋ|𝒪) [   P_θ(ℋ|𝒪)  -  P_θ+Δθ(ℋ|𝒪)   ]  
    ≈      - 1/2Δθ^T  E_P_θ(ℋ|𝒪) [  ∇^2    P_θ(ℋ|𝒪) ]Δθ


In (<ref>), the first order term is dropped as it equates to zero by (<ref>),  Hence, locally the KL divergence can be approximated  by the bilinear form:


    KL  (P_θ(ℋ| 𝒪) P_θ+Δθ(ℋ| 𝒪))  ≈   - 1/2Δθ^T  E_P_θ(ℋ| 𝒪) [  ∇^2    P_θ(ℋ|𝒪) ]Δθ

For discriminative models P_θ(ℋ|𝒪),  the Fisher Information  for a random variable θ  corresponds to  the expected  outer product of  score of the likelihood:


    I_θ   =    E_P_θ(ℋ|𝒪) [ ( ∇P_θ(ℋ|𝒪)  ) ( ∇P_θ (ℋ|𝒪)  )^T  ]


 In scenarios where  (<ref>) holds, I_θ  can be shown to be equal to the negative of the expectation of the Hessian w.r.t the distribution P_θ(ℋ|𝒪). Thus by substituting the expression of I_θ into (<ref>), the KL divergence measure is locally equivalent to the inner product:

    KL  (P_θ(ℋ|𝒪) P_θ+Δθ(ℋ|𝒪)) ≈1/2Δθ^T I_θΔθ


Let  α : (a, b)  ⊂ℝ→  X be a continuous map on X.  Assuming that the curve is regular i.e dα is injective at every point t ∈(a,b), the arc length of the curve corresponds to:

    s(t)    = _a^b ⟨ dα(t) , dα(t) ⟩_I_θ dt

If the choice of  I_θ corresponds to the Fisher Information matrix,  length of paths traversed in the parameter manifold can be interpreted as  local changes in the KL divergence.



§  ADAPTING THE EMPIRICAL FISHER TO  A YIELD  A PROPER RIEMANNIAN METRIC  


To recap, a Riemannian metric on a smooth parameter manifold X (see Appendix <ref>) is  a smooth map that assigns to each θ∈ X an inner product  I_θ  in T_θ X. As   Î_θ  is real and symmetric, by the  spectral decomposition theorem <cit.>, there exists a unitary basis of eigenvectors w.r.t the matrix becomes diagonalisable:

    Î_θ≡ V_θΛ_θ V^T_θ
 
where   V_θ is a square matrix whose i-th column  corresponds to the i-th eigenvector of Î_θ,  and Λ_θ is the diagonal matrix whose non-zero entries represent the associated eigenvalues. It should be noted that the entries of these two matrices  are functions of θ. To  keep the  notation uncluttered, the dependency on θ will be dropped for the remainder of this section whenever any of the individual factors in V_θΛ_θ V^T_θ is mentioned. Since Î_θ  is only guaranteed to be positive semi-definite, there will  exist  zero diagonal entries in Λ resulting in its rank being m where m< D (the dimensionality of the parameter space).  Under such circumstances, Î_θ^-1 will not exist and  it will no longer be possible to endow X with Riemannian metric of the form Î_θ^-1. To address this issue, this section derives an alternative metric to Î_θ that has the same structure and properties as the damped FI matrix but is guaranteed to be positive definite. From a high level perspective, the construction of the proposed  metric is achieved in two stages.

Step 1: partition the tangent space T_θ  X into  two disjoint subspaces such that one  subspace is spanned by eigenvectors of Î_θ associated with  non-zero eigenvalues.
 
 By re-arranging the columns of V such that the eigenvectors associated with the non-zero eigenvalues occupy positions within the first m  columns,  the image space of   Î_θ can be then essentially  captured by the matrix V_:,1:mΛ_1:m,1:m V^T_:,1:m. 

Let  ϕ be a map from X to ℝ^m given by 

    ϕ (θ) =  V^T_:,1:mθ


By  the Replacement theorem <cit.>, such a map can be shown to be maximal i.e the derivative dϕ is of full rank.  Under the Implicit Function theorem <cit.>, there exists a chart h (see Appendix <ref>)  on  X and a neighbourhood  𝒱 of  h(θ) such that  ϕ∘  h^-1 |_𝒱 = π|_𝒱 with  π : ℝ^D →ℝ^m being the projection map. It is easy to see that  from the  definition of ϕ that such a chart does exist and corresponds to V_:,1:m. Thus, within the open neighbourhood  𝒱, the derivative of ϕ∘  h^-1  corresponds to the linear map:

    d( ϕ∘  h^-1)(v)=  [ [     I_m × m 0_(m+1) × D;             ] ] ( v)

With respect to the map ϕ, the tangent space T_θ X can thus be expressed the disjoint sum: 

    T_θ_k X = T_ϕ∘  h^-1(θ)ℝ^m ⊕ker(d ( ϕ∘  h^-1) )

where  ker(d ( ϕ∘  h^-1) ) denotes the null space[ the ker of a linear map L : N → W between two vector spaces N and W, is the set of all elements v∈ N for which L(v) = 0]. By the above construction,  T_ϕ∘  h^-1(θ)ℝ^m can  now be identified  as a subspace of T_θ X. 


 Step 2: assign the identity matrix scaled  by a very small number ϵ to the  tangent subspace captured by the  kernel of d ( ϕ∘  h^-1). As the entries in the  diagonal  of Λ_1:m,1:m  are  functions of θ, endowing  T_ϕ∘  h^-1 (θ)ℝ^m with the inner product  Λ_1:m,1:m is equivalent to assigning a Riemannian metric to the associated subspace in T_θ X. Together,  the tangent space of X  can now be assigned   with the  following Riemannian metric:
 
    [
    [        Λ_1:m,1:,m                 0;                 0 ϵ I_(D-m) × (D-m);                   ]]

Switching back to the original basis coordinates, the proposed  metric then takes the particular form:

    V [
    [         Λ_1:m,1:m                 0;                 0 ϵ I_(D-m) × (D-m);                   ]] V^T  = Ĩ_θ


It can be seen that by construction  Ĩ_θ  corresponds to a positive definite matrix whose  image space is the direct sum of the image  and  the kernel space of the empirical Fisher matrix Î_θ. Apart from being a proper Riemannian metric, using CG to solve the appropriate linear system  Ĩ_θΔθ = -∇ F(θ_k ), has one particular advantage: the very first directions explored by the CG algorithm constitute to directions in the image space of Î_θ <cit.> i.e  directions considered important by the empirical Fisher are  traversed first during the initial stages of a CG run.  







IEEEtran
10



Amari1998a
S. Amari,
“Natural Gradient Works Efficiently in Learning,”
Proc. Neural Information Processing Systems (NIPS), 1998.

Amari1997
S. Amari,
“Neural Learning in Structured Parameter Spaces–Natural Riemannian Gradient,”
Proc. Advances in Neural Information Processing
  Systems (NIPS), 1997, pp. 127–133.
  Adnan2017
  A. Haider & P. Woodland,
 “Sequence Training of DNN Acoustic Models with Natural Gradient,"
IEEE Workshop on Automatic Speech Recognition and
 Understanding (ASRU), 
Dec. 2017, pp.178-183

Bottou2016
L. Bottou, F. Curtis & J. Nocedal,
Optimization Methods for Large-Scale Machine Learning,
arXiv preprint arXiv:1606.04838, 2016.

 Pascanu2013a
R. Pascanu &Y. Bengio,
“Revisiting Natural Gradient for Deep Networks",
arXiv preprint arXiv:1301.3584, jan 2013.

Desjardins2015
G. Desjardins, K. Simonyan & R. Pascanu,
“Natural Neural Networks,”
Proc. Advances in Neural Information Processing Systems
  (NIPS), 2015.

AmariBook
S. Amari,
“ Information geometry and its applications",
Springer 2016, 
ch 1, pp 3-27.

NocedalBook
J.  Nocedal & S.  Wright,
“ Numerical Optimization",
Springer 2016, 

Friedberg
 S. H.   Friedbergand   & J.A . Insel  & E.L.   Spence
 “ Linear Algebra-4th Edition",
Prentice Hall, 2002, 



kline1998calculus
 M. Kline
 “ Calculus: An Intuitive and Physical Approach",
Dover Publicatipns, 1998, 






