Title: Depth creates no more spurious local minima,
Abstract: We show that for any convex differentiable loss function, a deep linear
network has no spurious local minima as long as it is true for the two layer
case. When applied to the quadratic loss, our result immediately implies the
powerful result in [Kawaguchi 2016] that there is no spurious local minima in
deep linear networks. Further, with the recent work [Zhou and Liang 2018], we
can remove all the assumptions in [Kawaguchi 2016]. Our proof is short and
elementary. It builds on the recent work of [Laurent and von Brecht 2018] and
uses a new rank one perturbation argument.