Title: A Robust Multi-Batch L-BFGS Method for Machine Learning,
Abstract: This paper describes an implementation of the L-BFGS method designed to deal
with two adversarial situations. The first occurs in distributed computing
environments where some of the computational nodes devoted to the evaluation of
the function and gradient are unable to return results on time. A similar
challenge occurs in a multi-batch approach in which the data points used to
compute function and gradients are purposely changed at each iteration to
accelerate the learning process. Difficulties arise because L-BFGS employs
gradient differences to update the Hessian approximations, and when these
gradients are computed using different data points the updating process can be
unstable. This paper shows how to perform stable quasi-Newton updating in the
multi-batch setting, studies the convergence properties for both convex and
nonconvex functions, and illustrates the behavior of the algorithm in a
distributed computing platform on binary classification logistic regression and
neural network training problems that arise in machine learning.