Convergence of the Gauss–Newton method is not guaranteed in all instances. The approximation that needs to hold to be able to ignore the second-order derivative terms may be valid in two cases, for which convergence is to be expected: The function values are small in magnitude, at least around the minimum. The functions are only "mildly" nonlinear, so that is relatively small in magnitude. Improved versions With the Gauss–Newton method the sum of squares of the residuals S may not decrease at every iteration. However, since Δ is a descent direction, unless is a stationary point, it holds that for all sufficiently small . Thus, if divergence occurs, one solution is to employ a fraction of the increment vector Δ in the updating formula: In other words, the increment vector is too long, but it still points "downhill", so going just a part of the way will decrease the objective function S. An optimal value for can be found by using a line search algorithm, that is, the magnitude of is determined by finding the value that minimizes S, usually using a direct search method in the interval or a backtracking line search such as Armijo-line search. Typically, should be chosen such that it satisfies the Wolfe conditions or the Goldstein conditions. In cases where the direction of the shift vector is such that the optimal fraction α is close to zero, an alternative method for handling divergence is the use of the Levenberg–Marquardt algorithm, a trust region method. The normal equations are modified in such a way that the increment vector is rotated towards the direction of steepest descent, where D is a positive diagonal matrix. Note that when D is the identity matrix I and , then , therefore the direction of Δ approaches the direction of the negative gradient . The so-called Marquardt parameter may also be optimized by a line search, but this is inefficient, as the shift vector must be recalculated every time is changed. A more efficient strategy is this: When divergence occurs, increase the Marquardt parameter until there is a decrease in S. Then retain the value from one iteration to the next, but decrease it if possible until a cut-off value is reached, when the Marquardt parameter can be set to zero; the minimization of S then becomes a standard Gauss–Newton minimization. Large-scale optimization For large-scale optimization, the Gauss–Newton method is of