Title: Behavior of Accelerated Gradient Methods Near Critical Points of Nonconvex Functions,
Abstract: We examine the behavior of accelerated gradient methods in smooth nonconvex
unconstrained optimization, focusing in particular on their behavior near
strict saddle points. Accelerated methods are iterative methods that typically
step along a direction that is a linear combination of the previous step and
the gradient of the function evaluated at a point at or near the current
iterate. (The previous step encodes gradient information from earlier stages in
the iterative process.) We show by means of the stable manifold theorem that
the heavy-ball method method is unlikely to converge to strict saddle points,
which are points at which the gradient of the objective is zero but the Hessian
has at least one negative eigenvalue. We then examine the behavior of the
heavy-ball method and other accelerated gradient methods in the vicinity of a
strict saddle point of a nonconvex quadratic function, showing that both
methods can diverge from this point more rapidly than the steepest-descent
method.