JAWS: Predictive Inference Under Covariate Shift
share
We propose JAWS, a series of wrapper methods for distribution-free uncertainty quantification tasks under covariate shift, centered on our core method JAW, the JAckknife+ Weighted with likelihood-ratio weights. JAWS also includes computationally efficient Approximations of JAW using higher-order influence functions: JAWA. Theoretically, we show that JAW relaxes the jackknife+'s assumption of data exchangeability to achieve the same finite-sample coverage guarantee even under covariate shift. JAWA further approaches the JAW guarantee in the limit of either the sample size or the influence function order under mild assumptions. Moreover, we propose a general approach to repurposing any distribution-free uncertainty quantification method and its guarantees to the task of risk assessment: a task that generates the estimated probability that the true label lies within a user-specified interval. We then propose JAW-R and JAWA-R as the repurposed versions of proposed methods for Risk assessment. Practically, JAWS outperform the state-of-the-art predictive inference baselines in a variety of biased real world data sets for both interval-generation and risk-assessment auditing tasks.
READ FULL TEXT