Learning Metrics that Maximise Power for Accelerated A/B-Tests

Online controlled experiments are a crucial tool to allow for confident decision-making in technology companies. A North Star metric is defined (such as long-term revenue or user retention), and system variants that statistically significantly improve on this metric in an A/B-test can be considered superior. North Star metrics are typically delayed and insensitive. As a result, the cost of experimentation is high: experiments need to run for a long time, and even then, type-II errors (i.e. false negatives) are prevalent. …

companies cs.ir cs.lg decision long-term making metrics north star power retention revenue star stat.ap stat.ml technology test tests tool variants