the levenberg-marquardt algorithm is not suitable for networks with large number of parametersbecause the cost of the matrix computations becomes prohibitive for instanceobtaining the levenberg-marquardt search direction in ( usually incurs an ( cost in additionthe levenberg-marquardt algorithm is applicable only when we wish to train the network using the squared-error loss both of these shortcomings are mitigated to an extent with the quasi-newton or adaptive gradient methods described next limited-memory bfgs method all the methods discussed so far have been first-order optimization methodsthat ismetht (tht at the current (and/or immediate pastcanods that only use the gradient vector ut :th didate solution tht in trying to design more efficient second-order optimization methodwe may be tempted to use newton' method with search direction- - ut quasi-newton method where ht is the matrix of second-order partial derivatives of ` ( (th)at tht there are two problems with this approach firstwhile the computation of ut via algorithm typically costs ( )the computation of ht costs ( secondeven if we have somehow computed ht very fastcomputing the search direction - ut still incurs an ( cost both of these considerations make newton' method impractical for large insteada practical alternative is to use quasi-newton methodin which we directly aim to approximate - via matrix ct that satisfies the secant conditionct where dt :tht tht- and :ut ut- an ingenious formula that generates suitable sequence of approximating matrices {ct (each satisfying the secant conditionis the bfgs updating formula ( )which can be written as the recursion (see exercise )( ct ut > ct- ut > ut dt > ut :( > dt )- limited memory bfgs this formula allows us to update ct- to ct and then compute ct ut in ( time while this quasi-newton approach is better than the ( cost of newton' methodit may be still too costly in large-scale applications insteadan approximate or limited memory bfgs updating can be achieved in (dtime the idea is to store few of the most recent pairs {dt in order to evaluate its action on vector ut without explicitly constructing and storing ct in computer memory this is possiblebecause updating to in ( requires only the pair and similarly computing ct from only requires the history of the updates dt which can be shown as follows define the matrices at via the backward recursion )at :ia - : > and observe that all matrix vector productsa = for can be computed efficiently via the backward recursion starting with qt ut : > - tt (