Title: Neural Networks as Interacting Particle Systems: Asymptotic Convexity of the Loss Landscape and Universal Scaling of the Approximation Error,
Abstract: Neural networks, a central tool in machine learning, have demonstrated
remarkable, high fidelity performance on image recognition and classification
tasks. These successes evince an ability to accurately represent high
dimensional functions, potentially of great use in computational and applied
mathematics. That said, there are few rigorous results about the representation
error and trainability of neural networks. Here we characterize both the error
and the scaling of the error with the size of the network by reinterpreting the
standard optimization algorithm used in machine learning applications,
stochastic gradient descent, as the evolution of a particle system with
interactions governed by a potential related to the objective or "loss"
function used to train the network. We show that, when the number $n$ of
parameters is large, the empirical distribution of the particles descends on a
convex landscape towards a minimizer at a rate independent of $n$. We establish
a Law of Large Numbers and a Central Limit Theorem for the empirical
distribution, which together show that the approximation error of the network
universally scales as $O(n^{-1})$. Remarkably, these properties do not depend
on the dimensionality of the domain of the function that we seek to represent.
Our analysis also quantifies the scale and nature of the noise introduced by
stochastic gradient descent and provides guidelines for the step size and batch
size to use when training a neural network. We illustrate our findings on
examples in which we train neural network to learn the energy function of the
continuous 3-spin model on the sphere. The approximation error scales as our
analysis predicts in as high a dimension as $d=25$.