Title: Constrained Bayesian Optimization with Noisy Experiments,
Abstract: Randomized experiments are the gold standard for evaluating the effects of
changes to real-world systems. Data in these tests may be difficult to collect
and outcomes may have high variance, resulting in potentially large measurement
error. Bayesian optimization is a promising technique for efficiently
optimizing multiple continuous parameters, but existing approaches degrade in
performance when the noise level is high, limiting its applicability to many
randomized experiments. We derive an expression for expected improvement under
greedy batch optimization with noisy observations and noisy constraints, and
develop a quasi-Monte Carlo approximation that allows it to be efficiently
optimized. Simulations with synthetic functions show that optimization
performance on noisy, constrained problems outperforms existing methods. We
further demonstrate the effectiveness of the method with two real-world
experiments conducted at Facebook: optimizing a ranking system, and optimizing
server compiler flags.