Title: Structured Black Box Variational Inference for Latent Time Series Models,
Abstract: Continuous latent time series models are prevalent in Bayesian modeling;
examples include the Kalman filter, dynamic collaborative filtering, or dynamic
topic models. These models often benefit from structured, non mean field
variational approximations that capture correlations between time steps. Black
box variational inference with reparameterization gradients (BBVI) allows us to
explore a rich new class of Bayesian non-conjugate latent time series models;
however, a naive application of BBVI to such structured variational models
would scale quadratically in the number of time steps. We describe a BBVI
algorithm analogous to the forward-backward algorithm which instead scales
linearly in time. It allows us to efficiently sample from the variational
distribution and estimate the gradients of the ELBO. Finally, we show results
on the recently proposed dynamic word embedding model, which was trained using
our method.