algorithm limited-memory bfgs update inputbfgs history list { }hj= initial and input outputd -ch uwhere ct ut dt > ct- ut > ut dt > for hh do /backward recursion to compute - ui di ti > ui ti for do ui (ti > qdi /compute ( /compute recursion ( return -qthe value of -ch in summarya quasi-newton algorithm with limited-memory bfgs updating reads as follows algorithm quasi-newton minimization with limited-memory bfgs inputtraining set {(xi yi )}ni= initial weight matrices and bias vectors {wl bl } = =th activation functions {sl } = and history parameter outputthe parameters of the trained learner ut- /initialization while stopping condition is not met do (tht via algorithm compute `value ` ( (tht )and ut th ut ut- add ( to the bfgs history as the newest bfgs pair if the number of pairs in the bfgs history is greater than then remove the oldest pair from the bfgs history compute via algorithm using the bfgs historyc iand ut - while ` ( (tht )`value - dut do / /line-search along quasi-newton direction -ad tht+ tht - + return tht as the minimizer of the training loss adaptive gradient methods recall that the limited-memory bfgs method in the previous section determines search direction using the recent history of previously computed gradients {ut and input parameters {tht this is because the bfgs pairs {dt can be easily constructed from the identitiesdt tht tht- and ut ut- in other wordsusing only past gradient computations and with little extra computationit is possible to infer some of the second-order information