Title: Applying the Delta method in metric analytics: A practical guide with novel ideas,
Abstract: During the last decade, the information technology industry has adopted a
data-driven culture, relying on online metrics to measure and monitor business
performance. Under the setting of big data, the majority of such metrics
approximately follow normal distributions, opening up potential opportunities
to model them directly without extra model assumptions and solve big data
problems via closed-form formulas using distributed algorithms at a fraction of
the cost of simulation-based procedures like bootstrap. However, certain
attributes of the metrics, such as their corresponding data generating
processes and aggregation levels, pose numerous challenges for constructing
trustworthy estimation and inference procedures. Motivated by four real-life
examples in metric development and analytics for large-scale A/B testing, we
provide a practical guide to applying the Delta method, one of the most
important tools from the classic statistics literature, to address the
aforementioned challenges. We emphasize the central role of the Delta method in
metric analytics by highlighting both its classic and novel applications.