Title: DADAM: A Consensus-based Distributed Adaptive Gradient Method for Online Optimization,
Abstract: Adaptive gradient-based optimization methods such as ADAGRAD, RMSPROP, and
ADAM are widely used in solving large-scale machine learning problems including
deep learning. A number of schemes have been proposed in the literature aiming
at parallelizing them, based on communications of peripheral nodes with a
central node, but incur high communications cost. To address this issue, we
develop a novel consensus-based distributed adaptive moment estimation method
(DADAM) for online optimization over a decentralized network that enables data
parallelization, as well as decentralized computation. The method is
particularly useful, since it can accommodate settings where access to local
data is allowed. Further, as established theoretically in this work, it can
outperform centralized adaptive algorithms, for certain classes of loss
functions used in applications. We analyze the convergence properties of the
proposed algorithm and provide a dynamic regret bound on the convergence rate
of adaptive moment estimation methods in both stochastic and deterministic
settings. Empirical results demonstrate that DADAM works also well in practice
and compares favorably to competing online optimization methods.