procedure may diverge numerically within a few iterations. In contrast, implicit stochastic gradient descent (shortened as ISGD) can be solved in closed-form as: This procedure will remain numerically stable virtually for all as the learning rate is now normalized. Such comparison between classical and implicit stochastic gradient descent in the least squares problem is very similar to the comparison between least mean squares (LMS) and normalized least mean squares filter (NLMS). Even though a closed-form solution for ISGD is only possible in least squares, the procedure can be efficiently implemented in a wide range of models. Specifically, suppose that depends on only through a linear combination with features , so that we can write , where may depend on as well but not on except through . Least squares obeys this rule, and so does logistic regression, and most generalized linear models. For instance, in least squares, , and in logistic regression , where is the logistic function. In Poisson regression, , and so on. In such settings, ISGD is simply implemented as follows. Let , where is scalar. Then, ISGD is equivalent to: The scaling factor can be found through the bisection method since in most regular models, such as the aforementioned generalized linear models, function is decreasing, and thus the search bounds for are . Momentum Further proposals include the momentum method, which appeared in Rumelhart, Hinton and Williams' paper on backpropagation learning. Stochastic gradient descent with momentum remembers the update at each iteration, and determines the next update as a linear combination of the gradient and the previous update: that leads to: where the parameter which minimizes is to be estimated, is a step size (sometimes called the learning rate in machine learning) and is an exponential decay factor between 0 and 1 that determines the relative contribution of the current gradient and earlier gradients to the weight change. The name momentum stems from an analogy to momentum in physics: the weight vector , thought of as a particle traveling through parameter space, incurs acceleration from the gradient of the loss ("force"). Unlike in classical stochastic gradient descent, it tends to keep traveling in the same direction, preventing oscillations. Momentum has been used successfully by computer scientists in the training of artificial neural networks for several decades. The momentum method is closely related to underdamped Langevin dynamics, and may be combined with Simulated Annealing. Averaging Averaged stochastic