Feature Attribution

Definition

An explanation algorithm receives a point $x_0$ and the decision $f(x_0)$ of a model too complex to understand, queries $f$ (or its gradient), and returns an importance score per feature for this one decision. The alternative, interpretable ML, uses models simple enough to understand as a whole: decision trees, linear models, GAMs $f(x)={\sum }_i{f}_i(x_i)$.

Formula

Gradient-based explanation: rank the features by $|\partial f(x_0)/\partial x_i|$. SHAP: Shapley-weighted value differences. LIME: coefficients of a local linear surrogate.

Correlated features (the same feature twice, address and income) and interactions (XOR, two genes) make a single importance per feature ill-defined. Explanations can be manipulated off the data manifold, different algorithms disagree, and in adversarial settings (loan decisions) the provider can cherry-pick; explanation cards and interpretable models are the proposed answers.

Appears in

• Lecture 9.2, local post-hoc explanations
• Lecture 9.2, kinds of explanations
• Lecture 9.2, correlations and interactions
• Lecture 9.2, manipulation, different algorithms disagree

Related concepts

• SHAP
• LIME
• Counterfactual Explanation