, then also .) A more strict version requires also the reverse inequality: for a positive constant . For all n, . (This condition ensures that the directions of and are similar.) Lower bound for learning rates This addresses the question whether there is a systematic way to find a positive number - depending on the function f, the point and the descent direction - so that all learning rates satisfy Armijo's condition. When , we can choose in the order of , where is a local Lipschitz constant for the gradient near the point (see Lipschitz continuity). If the function is , then is close to the Hessian of the function at the point . See for more detail. Upper bound for learning rates In the same situation where , an interesting question is how large learning rates can be chosen in Armijo's condition (that is, when one has no limit on as defined in the section "Function minimization using backtracking line search in practice"), since larger learning rates when is closer to the limit point (if exists) can make convergence faster. For example, in Wolfe conditions, there is no mention of but another condition called curvature condition is introduced. An upper bound for learning rates is shown to exist if one wants the constructed sequence converges to a non-degenerate critical point, see : The learning rates must be bounded from above roughly by . Here H is the Hessian of the function at the limit point, is its inverse, and is the norm of a linear operator. Thus, this result applies for example when one uses Backtracking line search for Morse functions. Note that in dimension 1, is a number and hence this upper bound is of the same size as the lower bound in the section "Lower bound for learning rates". On the other hand, if the limit point is degenerate, then learning rates can be unbounded. For example, a modification of backtracking line search known as unbounded backtracking gradient descent (see ) allows the learning rate to be half the size , where is a constant. Experiments with simple functions such as show that unbounded backtracking gradient descent converges much faster than the basic version described in the section "Function minimization using backtracking line search in practice". Time efficiency An argument against the use of Backtracking line search, in particular in Large scale optimisation, is that