Title: Frequentist Consistency of Variational Bayes,
Abstract: A key challenge for modern Bayesian statistics is how to perform scalable
inference of posterior distributions. To address this challenge, variational
Bayes (VB) methods have emerged as a popular alternative to the classical
Markov chain Monte Carlo (MCMC) methods. VB methods tend to be faster while
achieving comparable predictive performance. However, there are few theoretical
results around VB. In this paper, we establish frequentist consistency and
asymptotic normality of VB methods. Specifically, we connect VB methods to
point estimates based on variational approximations, called frequentist
variational approximations, and we use the connection to prove a variational
Bernstein-von Mises theorem. The theorem leverages the theoretical
characterizations of frequentist variational approximations to understand
asymptotic properties of VB. In summary, we prove that (1) the VB posterior
converges to the Kullback-Leibler (KL) minimizer of a normal distribution,
centered at the truth and (2) the corresponding variational expectation of the
parameter is consistent and asymptotically normal. As applications of the
theorem, we derive asymptotic properties of VB posteriors in Bayesian mixture
models, Bayesian generalized linear mixed models, and Bayesian stochastic block
models. We conduct a simulation study to illustrate these theoretical results.