AI & ML / ML foundations / 08_interpretability.md

Interpretability

Updated 6 interview angles 4 min read source
On this page8
  1. Two different questions
  2. Global: start with permutation importance
  3. Local: SHAP
  4. Counterfactuals are what people actually want
  5. Glass-box models are still an option
  6. The trap in interviews
  7. Related
  8. Interview angle

Interpretability

“Why did the model say no?” is a product requirement in lending, insurance and hiring, a regulatory one under the EU AI Act, and a debugging tool everywhere else. It is also where a lot of candidates have only heard the word SHAP.

Two different questions

Question Called Answered by
Why this prediction? local SHAP, LIME, counterfactuals
What does the model use? global permutation importance, PDP

The distinction matters because they disagree, legitimately. A feature can be globally unimportant and decisive for one applicant. Answering a regulator’s “explain this decision” with a global importance chart is answering a different question.

Global: start with permutation importance

Shuffle one column, re-score, measure how much performance drops. Cheap, model-agnostic, and measured on held-out data, which is what makes it honest.

python 
from sklearn.inspection import permutation_importance

r = permutation_importance(
    model, X_val, y_val,
    n_repeats=10, random_state=0,
)

Gotcha: with correlated features, permutation importance splits the credit and understates both. Shuffle height when weight is also present and the model compensates, so neither looks important. Group correlated features and permute the group.

Avoid tree feature_importances_ (mean decrease in impurity) as your headline number: it is computed on training data and is biased toward high-cardinality and continuous features. It will tell you customer_id is your best feature.

Local: SHAP

SHAP assigns each feature a contribution to this prediction, with a property the alternatives lack: the contributions sum exactly to the difference between the prediction and the base rate.

python 
import shap

explainer = shap.TreeExplainer(model)
values = explainer.shap_values(X_row)

That additivity is why SHAP won. It makes an explanation you can put in front of a customer — “your application scored 0.31 against a base of 0.55; income contributed −0.12 and recent enquiries −0.09” — and the numbers reconcile.

Practical notes:

  • TreeExplainer is exact and fast for tree ensembles. KernelExplainer is model-agnostic and slow enough to be impractical past small samples.
  • Interventional vs conditional feature perturbation changes the answer when features are correlated. The default is usually fine; knowing the knob exists is the senior signal.
  • SHAP explains the model, not the world. It shows what the model used, which is not the same as what causes the outcome. See Classical ML and treat causal claims as a separate discipline.

LIME, and why SHAP mostly replaced it

LIME fits a simple local model around one point and reads its coefficients. It is intuitive and it is unstable — run it twice with different sampling and the explanation moves. For anything a customer or an auditor will see, that instability is disqualifying.

Counterfactuals are what people actually want

text 
"Declined."
   -> unsatisfying

"Declined. With £4,000 more annual income,
 this would have been approved."
   -> actionable

A counterfactual gives the smallest change that flips the decision. It is more useful than an attribution to a non-expert, and it is what “right to explanation” conversations usually converge on.

The constraint that makes it hard: the counterfactual must be actionable. “If you were 10 years younger” is not a recourse, it is evidence of a problem. Restrict the search to mutable features.

Glass-box models are still an option

If interpretability is a hard requirement, consider not needing post-hoc explanation at all:

Model Trade
Logistic regression fully readable, weaker fit
Decision tree, shallow readable rules, unstable
EBM / GAM near-boosting accuracy, readable

Explainable Boosting Machines and GAMs are the underused answer: additive by construction, so the “explanation” is the model rather than an approximation of it, and the accuracy gap to gradient boosting is often small on tabular data. Proposing one where the regulation is strict reads as senior.

The trap in interviews

Do not claim SHAP tells you causality, and do not claim an explanation makes a model safe. An explanation of a biased model is a well-documented biased model. Interpretability is a diagnostic and a compliance artefact; fairness is a separate measurement — see Fairness and bias.

Interview angle 6

  • “How would you explain a single decision?” - SHAP for the attribution, because the contributions sum exactly to the gap between the prediction and the base rate, so the explanation reconciles. Then a counterfactual, because “with £4,000 more income this would have been approved” is what a non-expert actually wants.
  • “Global or local importance?” - different questions that legitimately disagree. A feature can be globally unimportant and decisive for one applicant, so answering “explain this decision” with a global chart is answering something else.
  • “Why not use the tree’s feature_importances_?” - it is mean decrease in impurity, computed on training data and biased toward high-cardinality continuous features. It will rank an id column highly. Permutation importance on held-out data is the honest version.
  • “What breaks permutation importance?” - correlated features. Shuffling one lets the model compensate with its neighbour, so both look unimportant. Permute correlated features as a group.
  • “SHAP or LIME?” - SHAP, for stability and additivity. LIME fits a local surrogate and the explanation moves between runs, which is disqualifying for anything an auditor or customer sees.
  • “Does explaining a model make it fair?” - no. An explanation of a biased model is a well-documented biased model. Interpretability is a diagnostic; fairness is a separate measurement with its own metrics.
Previous
Hyperparameter tuning