A Variational Formula for Rényi Divergences

07/07/2020

∙

by   Jeremiah Birrell, et al.

∙

0

∙

share

We derive a new variational formula for the Rényi family of divergences, R_α(QP), generalizing the classical Donsker-Varadhan variational formula for the Kullback-Leibler divergence. The objective functional in this new variational representation is expressed in terms of expectations under Q and P, and hence can be estimated using samples from the two distributions. We illustrate the utility of such a variational formula by constructing neural-network estimators for the Rényi divergences.