in practical implementationwe keep single copy of in memory and apply the bfgs update to it at every iteration note that if the current is symmetricthen so is the updated matrix moreoverthe bfgs update is matrix of rank two since the kullback-leibler divergence is not symmetricit is possible to flip the roles of and in ( and instead solve min ( ca subject toa da adfp formula the solution (see exercise on page gives the davidon-fletcher-powell or dfp formula for updating the matrix from one iteration to the nextcdfp ddc { ( dfp update note that if the curvature condition holds and the current is symmetric positive definitethen so is its update example (diagonal hessian updatethe original bfgs formula requires ( storage and computationwhich may be unmanageable for large one way to circumvent the prohibitive quadratic cost is to only store and update diagonal hessian matrix from one iteration to the next if is diagonalthen we may not be able to satisfy the secant condition ( and maintain positive definiteness instead the secant condition ( can be relaxed to the set of inequalities - dwhich are related to the definition of subgradient for convex functions we can then find unique diagonal matrix by minimizing (xt xt+ awith respect to and subject to the constraints that and is diagonal the solution (exercise on page yields the updating formula for diagonal element ci of ci ci if di / ci ui ci ui ( ci / otherwisei where : (xt and we assume unit learning ratext+ xt au steepest descent example (scalar hessian updateif the identity matrix is used in place of the hessian in ( )one obtains steepest descent or gradient descent methodsin which the iteration ( reduces to xt+ xt at (xt the rationale for the name steepest descent is as follows if we start from any point and make an infinitesimal move in some directionthen the function value is reduced by the largest magnitude in the (unit normdirectionu: ( )/ ( ) this is seen from the following inequality for all unit vectors (that iskuk ) ( ( ut= = dt dt observe that equality is achieved if and only if uthis inequality is an easy consequence of the cauchy-schwarz inequality