downside with all newton-like methods is that at each step they require the calculation and inversion of an hessian matrixwhich has computing time of ( )and is thus infeasible for large one way to avoid this cost is to use quasi-newton methodsdescribed next quasi-newton methods the idea behind quasi-newton methods is to replace the inverse hessian in ( at iteration by an matrix satisfying the secant conditionc dsecant condition ( where xt xt- and (xt (xt- are vectors stored in memory at each iteration the secant condition is satisfiedfor exampleby the broyden' family of matricesa ( uu for some and since there is an infinite number of matrices that satisfy the condition ( )we need way to determine unique at each iteration such that computing and storing from one step to the next is fast and avoids any costly matrix inversion the following examples illustrate howstarting with an initial guess at such matrix can be efficiently updated from one iteration to the next example (low-rank hessian updatethe quadratic model ( can be strengthened by further assuming that exp( ( )is proportional to probability density that can be approximated in the neighborhood of xt by the pdf of the (xt+ - distribution this normal approximation allows us to measure the discrepancy between two pairs ( and ( using the kullback-leibler divergence between the pdfs of the - ( - and ( distributions (see exercise on page )- ( - : - tr( - ln | ( ( ( suppose that the latest approximation to the inverse hessian is and we wish to compute an updated approximation for step one approach is to find the symmetric matrix that minimizes its kullback-leibler discrepancy from cas defined abovesubject to the constraint ( in other wordsmin ( aa subject toa da athe solution to this constrained optimization (see exercise on page yields the broyden-fletcher-goldfarb-shanno or bfgs formula for updating the matrix from one iteration to the nextcbfgs dd ( ) ( { bfgs update ( bfgs formula