Title: ICLR Reproducibility Challenge Report (Padam : Closing The Generalization Gap Of Adaptive Gradient Methods in Training Deep Neural Networks),
Abstract: This work is a part of ICLR Reproducibility Challenge 2019, we try to
reproduce the results in the conference submission PADAM: Closing The
Generalization Gap of Adaptive Gradient Methods In Training Deep Neural
Networks. Adaptive gradient methods proposed in past demonstrate a degraded
generalization performance than the stochastic gradient descent (SGD) with
momentum. The authors try to address this problem by designing a new
optimization algorithm that bridges the gap between the space of Adaptive
Gradient algorithms and SGD with momentum. With this method a new tunable
hyperparameter called partially adaptive parameter p is introduced that varies
between [0, 0.5]. We build the proposed optimizer and use it to mirror the
experiments performed by the authors. We review and comment on the empirical
analysis performed by the authors. Finally, we also propose a future direction
for further study of Padam. Our code is available at:
this https URL