Title: Learning rate adaptation for federated and differentially private learning,
Abstract: We propose an algorithm for the adaptation of the learning rate for
stochastic gradient descent (SGD) that avoids the need for validation set use.
The idea for the adaptiveness comes from the technique of extrapolation: to get
an estimate for the error against the gradient flow which underlies SGD, we
compare the result obtained by one full step and two half-steps. The algorithm
is applied in two separate frameworks: federated and differentially private
learning. Using examples of deep neural networks we empirically show that the
adaptive algorithm is competitive with manually tuned commonly used
optimisation methods for differentially privately training. We also show that
it works robustly in the case of federated learning unlike commonly used
optimisation methods.