Title: Doubly Accelerated Stochastic Variance Reduced Dual Averaging Method for Regularized Empirical Risk Minimization,
Abstract: In this paper, we develop a new accelerated stochastic gradient method for
efficiently solving the convex regularized empirical risk minimization problem
in mini-batch settings. The use of mini-batches is becoming a golden standard
in the machine learning community, because mini-batch settings stabilize the
gradient estimate and can easily make good use of parallel computing. The core
of our proposed method is the incorporation of our new "double acceleration"
technique and variance reduction technique. We theoretically analyze our
proposed method and show that our method much improves the mini-batch
efficiencies of previous accelerated stochastic methods, and essentially only
needs size $\sqrt{n}$ mini-batches for achieving the optimal iteration
complexities for both non-strongly and strongly convex objectives, where $n$ is
the training set size. Further, we show that even in non-mini-batch settings,
our method achieves the best known convergence rate for both non-strongly and
strongly convex objectives.