Title: Online Learning Rate Adaptation with Hypergradient Descent,
Abstract: We introduce a general method for improving the convergence rate of
gradient-based optimizers that is easy to implement and works well in practice.
We demonstrate the effectiveness of the method in a range of optimization
problems by applying it to stochastic gradient descent, stochastic gradient
descent with Nesterov momentum, and Adam, showing that it significantly reduces
the need for the manual tuning of the initial learning rate for these commonly
used algorithms. Our method works by dynamically updating the learning rate
during optimization using the gradient with respect to the learning rate of the
update rule itself. Computing this "hypergradient" needs little additional
computation, requires only one extra copy of the original gradient to be stored
in memory, and relies upon nothing more than what is provided by reverse-mode
automatic differentiation.