Applying The Delta Method In Metric Analytics: A Practical Guide With Novel Ideas

Authors:
Alex Deng Microsoft
Ulf Knoblich Microsoft
Jiannan Lu Microsoft

Introduction:

This paper studies metrics to measure and monitor business performance. Motivated by fthe paper's real-life examples in metric development and analytics for large-scale A/B testing, the authors provide a practical guide to applying the Delta method, one of the most important tools from the classic statistics literature.

Abstract:

During the last decade, the information technology industry has adopted a data-driven culture, relying on online metrics to measure and monitor business performance. Under the setting of big data, the majority of such metrics approximately follow normal distributions, opening up potential opportunities to model them directly without extra model assumptions and solve big data problems via closed-form formulas using distributed algorithms at a fraction of the cost of simulation-based procedures like bootstrap. However, certain attributes of the metrics, such as their corresponding data generating processes and aggregation levels, pose numerous challenges for constructing trustworthy estimation and inference procedures. Motivated by four real-life examples in metric development and analytics for large-scale A/B testing, we provide a practical guide to applying the Delta method, one of the most important tools from the classic statistics literature, to address the aforementioned challenges. We emphasize the central role of the Delta method in metric analytics by highlighting both its classic and novel applications.

You may want to know: