Empirical Bayesian Selection for Value Maximization
share
We study the common problem of selecting the best m units from a set of n in the asymptotic regime m / n →α∈ (0, 1), where noisy, heteroskedastic measurements of the units' true values are available and the decision-maker wishes to maximize the average true value of the units selected. Given a parametric prior distribution, the empirical Bayesian decision rule incurs 𝒪_p(n^-1) regret relative to the Bayesian oracle that knows the true prior. More generally, if the error in the estimated prior is of order 𝒪_p(r_n), regret is 𝒪_p(r_n^2). In this sense selecting the best units is easier than estimating their values. We show this regret bound is sharp, by giving an example in which it is attained. Using priors calibrated from a dataset of over four thousand internet experiments, we find that empirical Bayes methods perform well in practice for detecting the best treatments given only a modest number of experiments.
READ FULL TEXT