Title: Inference on a New Class of Sample Average Treatment Effects,
Abstract: We derive new variance formulas for inference on a general class of estimands
of causal average treatment effects in a Randomized Control Trial (RCT). We
generalize Robins (1988) and show that when the estimand of interest is the
Sample Average Treatment Effect of the Treated (SATT or SATC for controls), a
consistent variance estimator exists. Although these estimands are equal to the
Sample Average Treatment Effect (SATE) in expectation, potentially large
differences in both accuracy and coverage can occur by the change of estimand,
even asymptotically. Inference on the SATE, even using a conservative
confidence interval, provides incorrect coverage of the SATT or SATC. We derive
the variance and limiting distribution of a new and general class of
estimands---any mixing between SATT and SATC---for which the SATE is a specific
case. We demonstrate the applicability of the new theoretical results using
Monte-Carlo simulations and an empirical application with hundreds of online
experiments with an average sample size of approximately one hundred million
observations per experiment. An R package, estCI, that implements all the
proposed estimation procedures is available.