hessian approximation is invertiblewe add suitable diagonal matrix to obtain the regularized version of the approximationt ui > ht = - + with this full-rank approximation of the hessianthe newton search direction in ( becomes- dt ui > ut ( = - + thusdt can be computed in ( ntime via the sherman-morrison algorithm further to thisthe search direction ( can be efficiently updated to the next one- + ui > ut+ dt+ = - + in ( ntimethus avoiding the usual ( ncost (see exercise on page nonlinear least squares consider the squared-error training loss in nonlinear regressionn ` ( ( ) ( (xi byi ) = where (bis nonlinear prediction function that depends on the parameter (for example( shows the nonlinear logistic prediction functionthe training loss can be written as kg( byk where ( :[ ( ) (xn )]is the vector of outputs we wish to minimize the training loss in terms of in the newton-like methods in section one derives an iterative minimization algorithm that is inspired by taylor expansion of ` ( ( )insteadgiven current guess bt we can consider the taylor expansion of the nonlinear prediction function gg( bg( bt gt ( bt ) where gt : (bt is the matrix of jacobi of ( bat bt denoting the residual et : ( bt and replacing ( bwith its taylor approximation in ` ( ( ))we obtain the approximation to the training loss in the neighborhood of bt ` ( ( ) gauss-newton gt ( bt et the minimization of the right-hand side is linear least-squares problem and therefore dt : bt satisfies the normal equationsg> gt dt > (-et assuming that > gt is invertiblethe normal equations yield the gauss-newton search directiondt -( > gt )- > et