Title: Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priors,
Abstract: We show that Entropy-SGD (Chaudhari et al., 2017), when viewed as a learning
algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior)
classifier, i.e., a randomized classifier obtained by a risk-sensitive
perturbation of the weights of a learned classifier. Entropy-SGD works by
optimizing the bound's prior, violating the hypothesis of the PAC-Bayes theorem
that the prior is chosen independently of the data. Indeed, available
implementations of Entropy-SGD rapidly obtain zero training error on random
labels and the same holds of the Gibbs posterior. In order to obtain a valid
generalization bound, we rely on a result showing that data-dependent priors
obtained by stochastic gradient Langevin dynamics (SGLD) yield valid PAC-Bayes
bounds provided the target distribution of SGLD is $\epsilon$-differentially
private. We observe that test error on MNIST and CIFAR10 falls within the
(empirically nonvacuous) risk bounds computed under the assumption that SGLD
reaches stationarity. In particular, Entropy-SGLD can be configured to yield
relatively tight generalization bounds and still fit real labels, although
these same settings do not obtain state-of-the-art performance.