Title: A framework for Multi-A(rmed)/B(andit) testing with online FDR control,
Abstract: We propose an alternative framework to existing setups for controlling false
alarms when multiple A/B tests are run over time. This setup arises in many
practical applications, e.g. when pharmaceutical companies test new treatment
options against control pills for different diseases, or when internet
companies test their default webpages versus various alternatives over time.
Our framework proposes to replace a sequence of A/B tests by a sequence of
best-arm MAB instances, which can be continuously monitored by the data
scientist. When interleaving the MAB tests with an an online false discovery
rate (FDR) algorithm, we can obtain the best of both worlds: low sample
complexity and any time online FDR control. Our main contributions are: (i) to
propose reasonable definitions of a null hypothesis for MAB instances; (ii) to
demonstrate how one can derive an always-valid sequential p-value that allows
continuous monitoring of each MAB test; and (iii) to show that using rejection
thresholds of online-FDR algorithms as the confidence levels for the MAB
algorithms results in both sample-optimality, high power and low FDR at any
point in time. We run extensive simulations to verify our claims, and also
report results on real data collected from the New Yorker Cartoon Caption
contest.