Quick Context
Standing on a hillside, "how steep is it?" has no single answer — it depends which way you face. A partial derivative answers it for one fixed direction; the gradient bundles those answers into a vector pointing straight up the slope.
Partials: Freeze Everything Else
To compute ∂f/∂x you treat y as a constant and differentiate normally. That is the whole idea — you are asking how f changes if you step east while refusing to move north.
f(x, y) = x² + y²
∂f/∂x = 2x (y held fixed)
∂f/∂y = 2y (x held fixed)
The blue and amber arrows on the plot are exactly these two directional slopes.
The Gradient Vector
Stack the partials and you get the gradient:
∇f = [ ∂f/∂x , ∂f/∂y ]
It has two remarkable properties, both visible on the plot:
- It points in the direction of steepest ascent — the fastest way up from where you stand.
- Its length is how steep that climb is. On a flat plateau the arrow shrinks to nothing.
Notice too that the gradient arrow always crosses the contour lines at right angles — contours are paths of constant height, so the steepest direction must be perpendicular to them.
Gradient Descent: Just Walk Backwards
Training a model means finding the lowest point of a loss surface. The gradient points uphill, so you step the other way:
x ← x − η · ∂f/∂x
y ← y − η · ∂f/∂y
Press Descend repeatedly and watch the point walk downhill, taking big strides where the surface is steep and tiny ones as it flattens out — automatic step-size control, straight out of the maths.
Real models have millions of parameters instead of two, so the gradient is a million-dimensional vector. The principle is identical; only the picture is impossible to draw.
Interactive Exploration Guide
- Click around the bowl. The gradient always points away from the centre — uphill — and shrinks as you approach the minimum.
- Press Run on the bowl. The path heads almost straight to the middle, decelerating as the gradient dies.
- Switch to the valley (x² + 4y²) and run again. The path zig-zags, because the surface is far steeper in y than in x — the exact pathology that Adam and momentum were invented to fix.
- Try the saddle and sit near the origin. One partial pushes up, the other down; the gradient nearly vanishes even though this is not a minimum — a genuine trap in high-dimensional optimisation.
- Raise the learning rate above 0.4 on the valley and run. The steps overshoot and the point diverges — the classic symptom of too large a learning rate.
Key Takeaway
Hold every other variable still to get a partial derivative; collect the partials to get the gradient. It points uphill, its length measures steepness, and stepping against it is what training a neural network literally means.