Feature relevance quantification in explainable AI: A causality problem
share
We discuss promising recent contributions on quantifying feature relevance using Shapley values, where we observed some confusion on which probability distribution is the right one for dropped features. We argue that the confusion is based on not carefully distinguishing between *observational* and *interventional* conditional probabilities and try a clarification based on Pearl's seminal work on causality. We conclude that *unconditional* rather than *conditional* expectations provide the right notion of *dropping* features in contradiction to the theoretical justification of the software package SHAP. Parts of SHAP are unaffected because unconditional expectations (which we argue to be conceptually right) are used as *approximation* for the conditional ones, which encouraged others to 'improve' SHAP in a way that we believe to be flawed. Further, our criticism concerns TreeExplainer in SHAP, which really uses conditional expectations (without approximating them by unconditional ones)
READ FULL TEXT