starting with an initial guess most minimization and root-finding algorithms create sequence using the iterative updating rulext+ xt at dt ( where at is (typically smallstep sizecalled the learning rateand the vector dt is the search direction at step the iteration ( continues until the sequence {xt is deemed to have converged to solutionor computational budget has been exhausted the performance of all such iterative methods depends crucially on the quality of the initial guess there are two broad categories of iterative optimization algorithms of the form ( )those of line search typewhere at iteration we first compute direction dt and then determine reasonable step size at along this direction for examplein the case of minimizationat may be chosen to approximately minimize (xt dt for fixed xt and dt those of trust region typewhere at each iteration we first determine suitable step size at and then compute an approximately optimal direction dt learning rate line search trust region in the following sectionswe review several widely-used root-finding and optimization algorithms of the line search type newton-like methods suppose we wish to find roots of function rn rn if is in we can approximate around point xt as (xf (xt (xt )( xt )where is the matrix of jacobi -the matrix of partial derivatives of see ( when (xt is invertiblethis linear approximation has root xt - (xt (xt this gives the iterative updating formula ( for finding roots of with direction dt - - (xt (xt and learning rate at this is known as newton' method (or the newton-raphson methodfor root-finding instead of unit learning ratesometimes it is more effective to use an at that satisfies the armijo inexact line search conditionk (xt at dt ) ( at (xt )kwhere is small heuristically chosen constantsay - for functionssuch an at always exists by continuity and can be computed as in the following algorithm newton' method armijo inexact line search