What the model does as one feature moves, and why the average of many individuals can describe none of them.
Overview
Partial dependence
A partial dependence plot answers: as this feature moves across its range, what happens to the guess on average?
The method is mechanical. Take the dataset. Set the feature to some value for *each* row, leaving all other columns untouched. Predict, and mean. Repeat across the feature's range, and plot.
It's model-agnostic and easy to read. This Is why it's all over.
Partial Dependence, ICE and SHAP
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
A partial dependence curve sweeps one feature across its range, holding the others as they are, and averages the predictions.
An ICE curve does the same for a single row, without averaging. One line per individual.
When the effect of a feature differs by subgroup, the average can be flat while no individual is.
SHAP answers a different question: how much did each feature contribute to this one prediction.
Partial Dependence, ICE and SHAP
Three ways to ask what a model is doing. The trap in the most common one.
The trap
Leave the interplay control on and look at the thick line. It's nearly flat, and the readout gives its slope as close to zero. Read on its own, it says this feature doesn't matter.
Now the ICE curves underneath tell a fully different story. each single curve has a steep slope. The feature matters enormously to each single row. It's just that it slopes up for one group and down for the other, and blending them cancels.
This isn't a contrived case. Any feature whose effect reverses by subgroup behaves like this: a treatment that helps one group and harms another, a price change that attracts some customers and repels others.
A flat partial dependence curve doesn't mean the feature is unimportant. It means the mean effect is zero. This Is a much weaker statement.
Switch the interplay off and the curves stack into a single shape. Now the mean shows each single well, and the partial dependence line is trustworthy.
ICE curves
Single if-based Expectation curves are partial dependence without the blending: one line per row, showing what the model predicts for *that* row as the feature sweeps.
They cost nothing extra, since partial dependence finds them and then throws them away. Plotting them is the standard defence against the trap above, and the diagnostic is simple: if the ICE curves are parallel, the partial dependence line is a fair summary. If they fan out or cross, it isn't, and there's an interplay worth finding.
The other assumption
Partial dependence sets a feature to a value for each row even so of whether that combination is possible. Sweeping age from 18 to 80 across a dataset creates rows with age 18 and forty years of work run into.
The model is asked to predict for combinations that never occur, and its answer there's extrapolation with nothing to anchor it. When features are strongly linked, part of a partial dependence curve is showing regions of input space the model has never seen.
Built up Local Effects plots address this by only varying a feature within the range of values that really co-occur with the other features' values.
SHAP
SHAP answers a different question. Partial dependence and ICE show the model across a range. SHAP shows one guess.
For a given row, it assigns each feature a number saying how much that feature pushed the guess away from the base value — the mean guess over the dataset. The numbers sum just to the difference:
prediction = base value + sum of every feature's SHAP value
That additivity is what makes it handy for an single explanation: it accounts for the whole guess, with nothing unattributed.
The values come from Shapley values in cooperative game theory, which distribute credit for a joint outcome fairly among contributors. finding them just needs testing each subset of features. This Is exponential, so setups rough — TreeSHAP just and quickly for trees, KernelSHAP by sampling for any of it else.
Answers
Scope
Partial dependence
how the prediction moves with a feature, on average
What makes a SHAP explanation trustworthy for one guess is that the parts sum to the whole. Each feature gets a value saying how far it pushed this guess from the base value — the mean guess over the dataset — and those pushes rebuild the guess just:
term
value
base value
0.30
age
+0.15
income
+0.10
region
−0.05
prediction
0.50
0.30 + 0.15 + 0.10 - 0.05 = 0.50, with nothing unattributed — that additivity is the property partial dependence and ICE don't have, and it's what lets SHAP explain a single row rather than the model's mean action. The values come from Shapley values in cooperative game theory, which distribute credit for a joint outcome fairly, and finding them just means testing each subset of features — exponential. This Is why TreeSHAP (exact, fast, for trees) and KernelSHAP (sampled, for any of it) exist. The caution is that additivity isn't causality: a model using postcode as a proxy for income will show postcode adding, which is a true statement about the model and not about the world.
The flat-curve trap is worth restating because it's silent: a partial dependence line near zero can hide single effects that are large but cancel — a feature that slopes up for one subgroup and down for another means to nothing, while mattering enormously to each row. Plotting the ICE curves underneath is the standard defence: parallel curves mean the mean is a fair summary, curves that fan out or cross mean there's an interplay and the mean is lying. Partial dependence also quietly assumes the feature can be swept on its own of the others, so with strongly linked features part of the curve is the model extrapolating into combinations that never occur — which is what Built up Local Effects plots were designed to avoid.
Where it goes wrong with Partial Dependence, ICE and SHAP
Reading a flat PD curve as "unimportant". Plot the ICE curves.
Skipping linked features. Part of the curve may be extrapolation.
Treating SHAP as causal. It fields a guess, not an outcome. A model that uses postcode as a proxy for income will show postcode adding, and that is a fact about the model.
Blending SHAP values into a global importance and stopping there. The mean abs SHAP value is a reasonable global summary, and it discards the direction and the interplays that made SHAP worth finding.
Questions people ask about Partial Dependence, ICE and SHAP
What does a partial dependence plot show? The average predicted outcome as one feature varies while the others keep their observed values. It shows the model's average response to that feature.
When is partial dependence misleading? When features are correlated, since averaging creates combinations that never occur in real data. ICE curves help reveal whether the average hides different behaviour.
What are ICE curves? One line per row showing how its prediction changes as a feature varies. Spread between the lines indicates interactions.
What is a SHAP value? The contribution of a feature to one prediction, relative to the average prediction. The contributions across features add up exactly to the difference.
Is SHAP the same as feature importance? Averaging the absolute SHAP values over a dataset gives a global importance. But SHAP also explains each row, which importance scores cannot.
Does SHAP show causation? No. It explains how the model uses its inputs, not how the world works. A feature can drive predictions simply because it correlates with the true cause.
One chart for the model, one number per row
Partial dependence shows the model's mean action. A Shapley value shows a single guess. Both are found from scratch here. Neither is a black box.
example_01.pyscikit-learn
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.inspection import partial_dependence
rng = np.random.default_rng(0)
n = 1500
X = rng.normal(size=(n, 4))
y = (3.0 * np.tanh(2 * X[:, 0]) + 1.5 * X[:, 1] + 2.0 * X[:, 0] * X[:, 2]
+ rng.normal(0, 0.4, n))
model = GradientBoostingRegressor(random_state=0).fit(X, y)
print("model R2: %.4f" % model.score(X, y))
print()
print("PARTIAL DEPENDENCE -- vary one feature, average over everything else.")
grid = np.linspace(-2, 2, 9)
print("%8s %12s %12s %12s" % ("value", "PD on x0", "PD on x1", "PD on x3"))
for v in grid:
row = []
for c in (0, 1, 3):
Xc = X.copy(); Xc[:, c] = v
row.append(model.predict(Xc).mean())
print("%8.2f %12.4f %12.4f %12.4f" % (v, row[0], row[1], row[2]))
print()
pd0 = partial_dependence(model, X, [0], grid_resolution=9, method="brute")
mine = []
for v in pd0["grid_values"][0]:
Xc = X.copy(); Xc[:, 0] = v
mine.append(model.predict(Xc).mean())
print("scikit-learn:", np.round(pd0["average"][0][:5], 4))
print("hand loop :", np.round(mine[:5], 4))
print(" identical, because that loop IS the definition.")
print()
print("x0 rises then flattens -- that is the tanh. x1 is a straight")
print("line. x3 is flat, because the model never used it.")
Output
The next block shows the averaging trap and why a flat PD line can hide real effects.
example_02.pyscikit-learn
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.inspection import partial_dependence
rng = np.random.default_rng(0)
n = 1500
X = rng.normal(size=(n, 4))
y = (3.0 * np.tanh(2 * X[:, 0]) + 1.5 * X[:, 1] + 2.0 * X[:, 0] * X[:, 2]
+ rng.normal(0, 0.4, n))
model = GradientBoostingRegressor(random_state=0).fit(X, y)
br = partial_dependence(model, X, [0], grid_resolution=9, method="brute")
rec = partial_dependence(model, X, [0], grid_resolution=9, method="recursion")
print("method='brute' re-predicts all rows. 'recursion' walks the trees:")
print(" brute :", np.round(br["average"][0][:5], 4))
print(" recursion:", np.round(rec["average"][0][:5], 4))
print(" recursion ignores how the other features are distributed.")
print()
print("the trap: partial dependence AVERAGES, so opposite effects cancel.")
print("x2 only matters through its interaction with x0:")
for v in (-2.0, 0.0, 2.0):
Xc = X.copy(); Xc[:, 2] = v
lo = X[:, 0] < 0
print(" x2=%+.0f -> overall PD %7.4f | x0<0 %7.4f, x0>0 %7.4f"
% (v, model.predict(Xc).mean(), model.predict(Xc)[lo].mean(),
model.predict(Xc)[~lo].mean()))
print(" the overall column barely moves while the two halves go in")
print(" opposite directions. a flat PD line does not mean unimportant.")
Output
The next block computes exact Shapley values for one row, showing how each feature contributes.
example_03.pyscikit-learn
import numpy as np
from itertools import combinations
from math import factorial
from sklearn.ensemble import GradientBoostingRegressor
rng = np.random.default_rng(0)
n = 1500
X = rng.normal(size=(n, 4))
names = ["x0", "x1", "x2", "x3"]
y = (3.0 * np.tanh(2 * X[:, 0]) + 1.5 * X[:, 1] + 2.0 * X[:, 0] * X[:, 2]
+ rng.normal(0, 0.4, n))
model = GradientBoostingRegressor(random_state=0).fit(X, y)
print("SHAPLEY VALUES -- explain ONE prediction, exactly.")
row = X[7]
base = model.predict(X).mean()
def value(subset):
Xc = X.copy()
for c in subset: Xc[:, c] = row[c]
return model.predict(Xc).mean()
shap = np.zeros(4)
for c in range(4):
others = [j for j in range(4) if j != c]
for size in range(4):
for sub in combinations(others, size):
w = factorial(size) * factorial(3 - size) / factorial(4)
shap[c] += w * (value(list(sub) + [c]) - value(list(sub)))
print(" the row:", np.round(row, 3))
print(" average prediction: %8.4f" % base)
for c in range(4):
print(" %s contributes %+8.4f" % (names[c], shap[c]))
print(" base + contributions: %8.4f" % (base + shap.sum()))
print(" model's prediction : %8.4f" % model.predict([row])[0])
print(" they match. contributions add up to exactly the gap between")
print(" this prediction and the average. no other method guarantees that.")
Output
The next block explains three different rows and shows the cost of exact computation.
example_04.pyscikit-learn
import numpy as np
from itertools import combinations
from math import factorial
from sklearn.ensemble import GradientBoostingRegressor
rng = np.random.default_rng(0)
n = 1500
X = rng.normal(size=(n, 4))
names = ["x0", "x1", "x2", "x3"]
y = (3.0 * np.tanh(2 * X[:, 0]) + 1.5 * X[:, 1] + 2.0 * X[:, 0] * X[:, 2]
+ rng.normal(0, 0.4, n))
model = GradientBoostingRegressor(random_state=0).fit(X, y)
print("%8s %10s %10s %10s %10s %12s" % ("row", *names, "prediction"))
for i in (7, 100, 900):
r = X[i]
sv = np.zeros(4)
for c in range(4):
others = [j for j in range(4) if j != c]
for size in range(4):
for sub in combinations(others, size):
w = factorial(size) * factorial(3 - size) / factorial(4)
Xa, Xb = X.copy(), X.copy()
for j in list(sub) + [c]: Xa[:, j] = r[j]
for j in sub: Xb[:, j] = r[j]
sv[c] += w * (model.predict(Xa).mean() - model.predict(Xb).mean())
print("%8d %10.3f %10.3f %10.3f %10.3f %12.4f" % (i, *sv, model.predict([r])[0]))
print()
print("x0's contribution changes sign between rows. PD gives one curve")
print("for the whole model; Shapley gives different numbers per row.")
print()
print("exact Shapley needs 2^k coalitions (16 here, >1M at k=20).")
print("TreeSHAP computes them exactly for tree models in polynomial time.")
print("KernelSHAP approximates by sampling coalitions.")
print()
print("both answer 'what did the MODEL do', not 'what causes what'. a")
print("feature can carry a large Shapley value because it proxies for")
print("something the model was never shown.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
A partial dependence curve is flat. What can you conclude?
Plotting ICE curves settles it: if they are parallel the average is a fair summary; if they fan out or cross, there is an interaction the average cancels.
What does the additivity of SHAP values give you?
That is what makes it usable as an explanation of a single prediction. Exact computation is exponential, so TreeSHAP and KernelSHAP approximate it.
Why can part of a partial dependence curve be untrustworthy when features are correlated?
Sweeping age across every row creates people aged 18 with forty years of experience. ALE plots avoid this by varying a feature only within ranges that actually occur.
Cheat sheet
Partial Dependence, ICE and SHAP
Plot partial dependence, ICE curves and SHAP values for one feature. See where partial dependence hides interactions the others reveal.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.