unlike the search direction ( for newton-like algorithmsthe search direction of gauss-newton algorithm does not require the computation of hessian matrix observe that in the gauss-newton approach we determine dt by viewing the search direction as coefficients in linear regression with feature matrix gt and response -et this suggests that instead of using linear regressionwe can compute dt via ridge regression with suitable choice for the regularization parameter dt -( > gt ngi )- > et if we replace ni with the diagonal matrix diag( > gt )we then obtain the levenbergmarquardt search directiondt -( > gt diag( > gt ))- > et ( recall that the ridge regularization parameter has the following effect on the least-squares solutionwhen it is zerothen the solution dt coincides with the search direction of the gauss-newton methodand when tends to infinitythen kdt tends to zero thusg controls both the magnitude and direction of vector dt simple version of the levenbergmarquardt algorithm is the following algorithm levenberg-marquardt for minimizing kg( byk inputan initial guess stopping error training set outputan approximate minimizer of kg( byk and (or another default value while stopping condition is not met do compute the search direction dt via ( et+ ( bt dt if ket+ ket then / et+ et bt+ bt dt else - + return bt constrained minimization via penalty functions constrained optimization problem of the form ( can sometimes be reformulated as simpler unconstrained problem -for examplethe unconstrained set can be transformed to the feasible region of the constrained problem via function ph rn rn such that ph(ythen( is equivalent to the minimization problem min (ph( ))yy in the sense that solution xof the original problem is obtained from transformed solution yvia xph(ytable lists some examples of possible transformations levenbergmarquardt