Hessian) is to add a scaled identity matrix to the Hessian, , with the scale adjusted at every iteration as needed. For large and small Hessian, the iterations will behave like gradient descent with step size . This results in slower but more reliable convergence where the Hessian doesn't provide useful information. Some caveats Newton's method, in its original version, has several caveats: It does not work if the Hessian is not invertible. This is clear from the very definition of Newton's method, which requires taking the inverse of the Hessian. It may not converge at all, but can enter a cycle having more than 1 point. See the section "Failure analysis" in Newton's method. It can converge to a saddle point instead of to a local minimum, see the section "Geometric interpretation" in this article. The popular modifications of Newton's method, such as quasi-Newton methods or Levenberg-Marquardt algorithm mentioned above, also have caveats: For example, it is usually required that the cost function is (strongly) convex and the Hessian is globally bounded or Lipschitz continuous, for example this is mentioned in the section "Convergence" in this article. If one looks at the papers by Levenberg and Marquardt in the reference for Levenberg–Marquardt algorithm, which are the original sources for the mentioned method, one can see that there is basically no theoretical analysis in the paper by Levenberg, while the paper by Marquardt only analyses a local situation and does not prove a global convergence result. One can compare with Backtracking line search method for Gradient descent, which has good theoretical guarantee under more general assumptions, and can be implemented and works well in practical large scale problems such as Deep Neural Networks. See also Quasi-Newton method Gradient descent Gauss–Newton algorithm Levenberg–Marquardt algorithm Trust region Optimization Nelder–Mead method Notes References External links Optimization algorithms and methods fr:Méthode de Newton