Title: Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints,
Abstract: Algorithm-dependent generalization error bounds are central to statistical
learning theory. A learning algorithm may use a large hypothesis space, but the
limited number of iterations controls its model capacity and generalization
error. The impacts of stochastic gradient methods on generalization error for
non-convex learning problems not only have important theoretical consequences,
but are also critical to generalization errors of deep learning.
In this paper, we study the generalization errors of Stochastic Gradient
Langevin Dynamics (SGLD) with non-convex objectives. Two theories are proposed
with non-asymptotic discrete-time analysis, using Stability and PAC-Bayesian
results respectively. The stability-based theory obtains a bound of
$O\left(\frac{1}{n}L\sqrt{\beta T_k}\right)$, where $L$ is uniform Lipschitz
parameter, $\beta$ is inverse temperature, and $T_k$ is aggregated step sizes.
For PAC-Bayesian theory, though the bound has a slower $O(1/\sqrt{n})$ rate,
the contribution of each step is shown with an exponentially decaying factor by
imposing $\ell^2$ regularization, and the uniform Lipschitz constant is also
replaced by actual norms of gradients along trajectory. Our bounds have no
implicit dependence on dimensions, norms or other capacity measures of
parameter, which elegantly characterizes the phenomenon of "Fast Training
Guarantees Generalization" in non-convex settings. This is the first
algorithm-dependent result with reasonable dependence on aggregated step sizes
for non-convex learning, and has important implications to statistical learning
aspects of stochastic gradient methods in complicated models such as deep
learning.