Title: A unified theory of adaptive stochastic gradient descent as Bayesian filtering,
Abstract: We formulate stochastic gradient descent (SGD) as a Bayesian filtering
problem. Inference in the Bayesian setting naturally gives rise to BRMSprop and
BAdam: Bayesian variants of RMSprop and Adam. Remarkably, the Bayesian approach
recovers many features of state-of-the-art adaptive SGD methods, including
amoungst others root-mean-square normalization, Nesterov acceleration and
AdamW. As such, the Bayesian approach provides one explanation for the
empirical effectiveness of state-of-the-art adaptive SGD algorithms.
Empirically comparing BRMSprop and BAdam with naive RMSprop and Adam on MNIST,
we find that Bayesian methods have the potential to considerably reduce test
loss and classification error.