0. Practicalities for using in the stochastic setting and implementation in deep neural networks The algorithm described above is for the deterministic setting, which as reported above has good theoretical guarantees. When one uses Armijo's algorithm in real life settings where random features play important roles, there are practicalities one needs to take care about. This section describes some main points to be noted in the more theoretical setting of stochastic optimization and the more realistic setting of mini-batch in deep neural networks. 1) Stochastic optimization: The typical statement of this setting is as follows: Assume that we have a function depending on a random variable . Now we want to find minimum of the expectation function , here computes the expectation with respect to the random variable . In general, it is very difficult to compute the expectation, in particular when working for example with deep neural networks (see the next part). Therefore, we can only do as follows: At step n: choose randomly variables (with respect to the given distribution), and replace the (unknown) function by an approximation . We need to modify Armijo's backtracking line search as follows: Then the descent direction is chosen as and the Armijo's learning rate is chosen relative to the function . It is noted that in general is allowed to change. There are works which let increase. The following example explains the motivation for such a scheme (also, see theoretical justification later). However, it will be explained in the section about implementing in deep neural networks that there are difficulties/not very good performance with implementing this scheme. Example 1: Consider the simple function , which is convex with a unique critical point . Recall the normal distribution . Then , where is a random variable with distribution . Then, one can check with Python, that if we choose and for all step n, and run Armijo's Backtracking line search with maximum learning rate , in this setting as described above, then we have divergence (not convergence as one would hope for this too simple convex function)! [What happens here is that, while small, there is a non-zero probability of the random value of is , in which case what Armijo's backtracking line search will do is to push away from the point 0 by a large amount.] Another way is, originating from the classical work , in SGD to keep constant