Goodhart’s Law and the Dangers of Metric Selection with A/B Testing

The surprisingly counterintuitive problems of choosing metrics (and how it relates to cobras)

Experimentation has many parts: choosing the hypothesis, implementing the variations, assignment, results analysis, documentation, etc, and each of these can have their own nuances that make it complicated. Usually choosing the metric (or metrics) that determines if your hypothesis is correct is straightforward. However there are some situations where choosing metrics can be problematic, and surface classic issues of Goodhart’s law. …

a-b-testing a/b testing b testing dangers data analysis data science law metrics statistics testing