Best of Three Worlds: Adaptive Experimentation for Digital Marketing in Practice

arXiv:2402.10870v1 Announce Type: new
Abstract: Adaptive experimental design (AED) methods are increasingly being used in industry as a tool to boost testing throughput or reduce experimentation cost relative to traditional A/B/N testing methods. However, the behavior and guarantees of such methods are not well-understood beyond idealized stationary settings. This paper shares lessons learned regarding the challenges of naively using AED systems in industrial settings where non-stationarity is prevalent, while also providing perspectives on the proper objectives and system specifications in …

abstract arxiv behavior best of beyond boost cost cs.lg design digital digital marketing experimental experimentation industry marketing practice reduce stat.me testing tool type