of these theoretical results are not satisfied (too strong to be satisfied) by the cost functions in deep neural networks. It is worthy to notice that current successful implementations of backtracking in deep neural networks use constant mini-batch sizes . However, the difference with Example 1 is that one does not choose k_n=1 since that is too small. Indeed, one has the following. Example 2. Let the function F(x) and the distribution be as in Example 1, where . If one chooses for all n, then one can check with Python that Armijo's backtracking line search with maximum learning rate in this stochastic setting converges! This can be explained that in this example, choosing for all n is not good enough to approximate the function F(x), but choosing for all n is a good approximation. The same happens with deep neural networks: one needs to choose a big enough mini-batch size depending on the specific question and setting (see next section). People who are familiar with undergraduate calculus can find an analogy in using Riemann sums to approximate an integral. 2) deep neural networks: While stochastic optimization is meant to be a theoretical model for what happens in training deep neural networks, in practice there are important differences in many aspects: assumptions, resources, ways to proceed and goals. Assumptions: in stochastic optimization, among many others, there is usually the requirement that the cost function is (strongly-)convex. This is hardly satisfied or verified in practice. Summary In summary, backtracking line search (and its modifications) is a method which is easy to implement, is applicable for very general functions, has very good theoretical guarantee (for both convergence to critical points and avoidance of saddle points) and works well in practice. Several other methods which have good theoretical guarantee, such as diminishing learning rates or standard GD with learning rate <1/L – both require the gradient of the objective function to be Lipschitz continuous, turn out to be a special case of Backtracking line search or satisfy Armijo's condition. Even though a priori one needs the cost function to be continuously differentiable to apply this method, in practice one can apply this method successfully also for functions which are continuously differentiable on a dense open subset such as or . See also Gradient descent Stochastic gradient descent Wolfe conditions References Mathematical optimization Optimization algorithms and methods