Title: Percent Change Estimation in Large Scale Online Experiments,
Abstract: Online experiments are a fundamental component of the development of
web-facing products. Given the large user-base, even small product improvements
can have a large impact on an absolute scale. As a result, accurately
estimating the relative impact of these changes is extremely important. I
propose an approach based on an objective Bayesian model to improve the
sensitivity of percent change estimation in A/B experiments. Leveraging
pre-period information, this approach produces more robust and accurate point
estimates and up to 50% tighter credible intervals than traditional methods.
The R package abpackage provides an implementation of the approach.