(even can choose to be always 1), but let the learning rates going to 0. [Even if the random value of is , since the learning rate is smaller and smaller, this scheme will push away from 0 less than backtracking.] While this scheme works well also for this example, again in the setting of deep neural networks the experimental results are not very good. It is worthy to note also that if one uses SGD and keeps constant, but also learning rates constant, then one can observe divergence even for some values of the learning rates for which there is convergence guarantee for . So what mentions in Example 1 is not the fault of backtracking itself, but also for SGD, the most studied optimization algorithm. Example 1'. The same function F(x) and distribution . Choose and for all step n. Note that the function F(x) is with . Hence, by classical theory, for learning rate , SGD with constant learning rate will converge for F(x). However, one check with Python that for the associated stochastic optimization problem, even with the choice of a smaller learning rate 0.1, one observes divergence to infinity! On the other hand, with the learning rate 0.05 then one observes convergence to the minimum 0. Therefore, it seems that even for this simple function, the choice of (constant) learning rate must take into account the distribution, and not just that from the deterministic function F(x), for to obtain convergence for the stochastic optimization problem. [Note: if here one runs Armijo's Backtracking line search with maximum learning rate , then one observes convergence! Hence, still Armijo's Backtracking line search helps with choosing a good learning rate to use, compared to SGD with constant learning rates.] A typical theoretical result (in the literature, one may find weaker results than what stated here, either because requiring stronger assumptions or proving weaker conclusions) in this setting is as follows: Prototype theorem: Let be convex (or strongly convex) and in . Then both of the following schemes guarantee convergence: Scheme 1: Backtracking with increasing "mini-batch" sizes . Scheme 2: SGD with learning rates decreasing to 0. Why these schemes yield not very good experimental results in deep neural networks can be because: - The implementation does not take into account the differences between stochastic optimization and deep neural networks (see more in the next section); and/or - The assumptions