Assign Experiment Variants at Scale in A/B tests

How we improved the randomized assignment algorithm for online experiments at Wish

Contributors: Qike (Max) Li, Samir Jamkhande

Recently at Wish, we discovered a sample ratio mismatch (SRM) in an A/A test. SRM refers to the mismatch between the sample ratio set by the experimenter and the observed sample ratio. SRM indicates potential data quality problems that change the outcomes of your experiments [1]. Microsoft shared an example in which the A/B …

a-b-testing data science experiment experimentation randomization scalability scale variants