contained in the hessian matrix of ` (thin addition to the bfgs methodthere are other ways in which we can exploit the history of past gradient computations one approach is to use the normal approximation methodin which the hessian of ` at tht is approximated via ui > ( ht = - + where ut- + ut are the most recently computed gradients and is tuning parameter (for exampleg /hthe search direction is then given by - - ut which can be computed quickly in ( dtime either using the qr decomposition (exercises and )or the sherman-morrison algorithm this approach requires that we store the last gradient vectors in memory another approach that completely bypasses the need to invert hessian approximation bt is the adaptive gradient or adagrad methodin which we only store the diagonal of and use the search directionb )- / ut -diag( adagrad we can avoid storing any of the gradient history by instead using the slightly different search direction -ut vt where the vector vt is updated recursively via vt vt- ut ut with this updating of vt the difference between the vector vt and the diagonal of will be negligible the hessian more sophisticated version of adagrad is the adaptive moment estimation or adam methodin which we not only average the vectors {vt }but also average the gradient vectors {ut }as follows algorithm updating of search direction at iteration via adam inputut ut- vt- tht and parameters (ahv hu )equal toe ( - outputut  tht+ ut - hu vt ut hv - hv  ut ut ( - - vt vt ( hv  tht+ tht ut * - return ut vt tht+ here we divide two vectors componentwise adam