A Study of Symbiosis Bias in A/B Tests of Recommendation Algorithms
share
One assumption underlying the unbiasedness of global treatment effect estimates from randomized experiments is the stable unit treatment value assumption (SUTVA). Many experiments that compare the efficacy of different recommendation algorithms violate SUTVA, because each algorithm is trained on a pool of shared data, often coming from a mixture of recommendation algorithms in the experiment. We explore, through simulation, cluster randomized and data-diverted solutions to mitigating this bias, which we call "symbiosis bias."
READ FULL TEXT