Universality laws for Gaussian mixtures in generalized linear models
Let (x_i, y_i)_i=1,โฆ,n denote independent samples from a general mixture distribution โ_cโ๐ฯ_cP_c^x, and consider the hypothesis class of generalized linear models ลท = F(ฮ^โคx). In this work, we investigate the asymptotic joint statistics of the family of generalized linear estimators (ฮ_1, โฆ, ฮ_M) obtained either from (a) minimizing an empirical risk Rฬ_n(ฮ;X,y) or (b) sampling from the associated Gibbs measure exp(-ฮฒ n Rฬ_n(ฮ;X,y)). Our main contribution is to characterize under which conditions the asymptotic joint statistics of this family depends (on a weak sense) only on the means and covariances of the class conditional features distribution P_c^x. In particular, this allow us to prove the universality of different quantities of interest, such as the training and generalization errors, redeeming a recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis. Finally, we discuss the applications of our results to different machine learning tasks of interest, such as ensembling and uncertainty
READ FULL TEXT