|{zcauchy-schwartz kuk uthe steepest descent iterationxt+ xt at (xt )still requires suitable choice of the learning rate at an alternative way to think about the iteration is to assume that the learning rate is always unityand that at each iteration we use an inverse hessian matrix of the form at for some positive constant at satisfying the secant condition ( with matrix of the form ai is not possible howeverit is possible to choose so that the secant condition ( is satisfied in the direction of (or alternatively dthis gives the barzilai-borwein formulas for the learning rate at iteration tkdk or alternatively at ( at normal approximation method let phht- ( xt+ denote the pdf of the (xt+ - distribution as we already saw in example the quadratic approximation ( of in the neighborhood of xt is equivalent (up to constantto the minus of the logarithm of the pdf phh- ( xt+ in other wordst we use phh- ( as simple model for the density + exp( ( )exp( ( )dy one consequence of the normal approximation is that for in the neighborhood of xt+ we can writef (xln phh- ( xt+ -ht ( xt+ in other wordsusing the fact that > ht ( ) ( )]ht ( xt+ )( xt+ )ht and taking expectations on both sides with respect to (xt+ - givese (xf ( )]ht this suggests thatgiven the gradient vectors computed in the past (where stands for historyof newton iterationsui : (xi ) ( )tthe hessian matrix ht can be approximated via the average ui > = - + shortcoming of this approximation is thatunless is large enoughthe hessian approxp imation ti= - + ui > may not be full rank and hence not invertible to ensure that the barzilaiborwein formulas