How to Select the Right Statistical Tests for Different A/B Metrics

A Discussion of the go-to methods for 5 Types of A/B Metrics

TL;DR

This article summarizes A/B test evaluation metrics into 5 categories and outlines the suggested statistics testing for its significance values in the table below.

Table of Contents

1. Why Should We Care
2. User Average Metrics
3. User-level Conversion Metrics
4. Pageview-level Conversion Metrics
5. Percentile Metrics
6. SUM Metrics
7. Summary and Practical Tips
8. Note

Why Should We Care?

While the t-tests are powerful, they are not universally applicable in …

a-b-testing data science deep-dives math metrics statistical statistics tests