Explainability in Practice: SHAP Values and When to Actually Use Them

Your model is accurate. Your stakeholders hate it anyway. They want to know why it made a decision, and "the weights are complex" isn't cutting it. This is where SHAP values enter the conversation—and where most teams go wrong.

SHAP (SHapley Additive exPlanations) values have become the explainability darling of machine learning. They're mathematically sound, theoretically elegant, and computationally expensive. Before you add them to your pipeline, you need to understand what they actually solve and where they create more problems than answers.

What SHAP Values Actually Do

SHAP values calculate each feature's contribution to a prediction by considering all possible orderings of features. The result: a number for each input that tells you how much that feature pushed the prediction away from the model's average output.

Here's a basic example with a tabular model:

pythonCopy
import shap
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

model = xgb.XGBClassifier(use_label_encoder=False, eval_metric='logloss')
model.fit(X_train, y_train)

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

shap.summary_plot(shap_values, X_test, plot_type="bar")

This gives you feature importance with direction and magnitude. That's genuinely useful. The problem is assuming it's always the right tool.

When SHAP Values Make Sense

High-Stakes, Black-Box Models

If you're using a gradient boosted model or deep neural network in healthcare, lending, or hiring decisions, SHAP values justify their computational cost. Regulators and courts increasingly demand explanations. SHAP's game-theoretic foundation holds up better than alternatives when someone questions your methodology.

Complex Feature Interactions

Permutation importance and coefficients break down when features correlate heavily. SHAP values handle this better because they account for feature dependencies. If your features are genuinely tangled, SHAP is one of the few methods that doesn't make things worse.

Debugging Model Behavior

When a model makes predictions that feel wrong, SHAP can pinpoint which features caused the error. This is invaluable for finding data quality issues or training data quirks. At LavaPi, we've used SHAP analysis to catch cases where models were memorizing artifacts rather than learning patterns.

When SHAP Values Are Overkill

Linear Models and Tree Ensembles with Few Features

If you're using logistic regression or a decision tree with <20 features, you probably don't need SHAP. Linear model coefficients are already interpretable. For trees, standard feature importance is fast and adequate.

Real-Time Prediction Systems

SHAP calculation adds latency. Computing exact SHAP values for a single prediction can take seconds or minutes depending on your model size. If you're making decisions in milliseconds, SHAP doesn't fit. Use approximations (KernelSHAP, TreeSHAP) carefully, and understand you're trading accuracy for speed.

When Domain Experts Can't Act on the Explanation

If your audience can't change behavior based on "feature X contributed +0.3 to the score," why compute it? Explanations only matter if they inform decisions. Sometimes a simpler, qualitative explanation works better.

The Real Question

Before implementing SHAP, ask: Who needs this explanation, and what will they do with it? If it's internal debugging, SHAP is strong. If it's end-user transparency, you might need something simpler. If it's regulatory compliance, SHAP is defensible.

SHAP values are a professional-grade tool. Use them professionally—which means understanding their limitations as well as their strengths. The best explanation is the one that actually gets used.