Title: On Scalable Inference with Stochastic Gradient Descent,
Abstract: In many applications involving large dataset or online updating, stochastic
gradient descent (SGD) provides a scalable way to compute parameter estimates
and has gained increasing popularity due to its numerical convenience and
memory efficiency. While the asymptotic properties of SGD-based estimators have
been established decades ago, statistical inference such as interval estimation
remains much unexplored. The traditional resampling method such as the
bootstrap is not computationally feasible since it requires to repeatedly draw
independent samples from the entire dataset. The plug-in method is not
applicable when there are no explicit formulas for the covariance matrix of the
estimator. In this paper, we propose a scalable inferential procedure for
stochastic gradient descent, which, upon the arrival of each observation,
updates the SGD estimate as well as a large number of randomly perturbed SGD
estimates. The proposed method is easy to implement in practice. We establish
its theoretical properties for a general class of models that includes
generalized linear models and quantile regression models as special cases. The
finite-sample performance and numerical utility is evaluated by simulation
studies and two real data applications.