With two inputs there is no single slope — it depends which way you walk. Freeze one variable at a time to get the partials, then combine them into the gradient: the arrow pointing straight uphill.
Controls
x2.0
y1.5
learning rate0.15
Brighter = higher. The green arrow is the gradient (uphill); descent steps the opposite way.
Contour Map of f(x, y)
click anywhere to move the point
Live Calculation
f(x, y)
0
∂f/∂x
0
∂f/∂y
0
|∇f|
0
Partial Derivatives and the Gradient
One slope per direction — and the vector that combines them into "uphill".
Start here
Standing on a hillside, "how steep is it?" has no single answer — it depends which way you face. A partial derivative answers it for one fixed direction; the gradient bundles those answers into a vector pointing straight up the slope.
Partials: Freeze Everything Else
To compute ∂f/∂x you treat y as a constant and differentiate normally. That is the whole idea — you are asking how f changes if you step east while refusing to move north.
The blue and amber arrows on the plot are exactly these two directional slopes.
The Gradient Vector
Stack the partials and you get the gradient:
It has two remarkable properties, both visible on the plot:
It points in the direction of steepest ascent — the fastest way up from where you stand.
Its length is how steep that climb is. On a flat plateau the arrow shrinks to nothing.
Notice too that the gradient arrow always crosses the contour lines at right angles — contours are paths of constant height, so the steepest direction must be perpendicular to them.
Gradient Descent: Just Walk Backwards
Training a model means finding the lowest point of a loss surface. The gradient points uphill, so you step the other way:
Press Descend repeatedly and watch the point walk downhill, taking big strides where the surface is steep and tiny ones as it flattens out — automatic step-size control, straight out of the maths.
Real models have millions of parameters instead of two, so the gradient is a million-dimensional vector. The principle is identical; only the picture is impossible to draw.
One variable at a time
A partial derivative asks a deliberately narrow question: if I change this input by a tiny amount and hold every other input still, how much does the output move?
For f(x, y) = x² + 3xy + y²:
Treat y as a constant: ∂f/∂x = 2x + 3y.
Treat x as a constant: ∂f/∂y = 3x + 2y.
At the point (2, 1) those give 4 + 3 = 7 and 6 + 2 = 8. So moving along x increases f at a rate of 7 per unit, and along y at 8 per unit.
That is all a partial derivative is — an ordinary derivative with the other variables frozen. The curly ∂ is only there to remind you that other variables exist.
The gradient: all the partials in one vector
Collect every partial derivative into a vector and you have the gradient:
∇f = [ ∂f/∂x, ∂f/∂y ] = [7, 8] at (2, 1)
Two properties make this the central object in machine learning:
The gradient points in the direction of steepest increase. Of all the directions you could step, that vector is the one where f grows fastest.
Its length is how steep that steepest direction is. A long gradient means a cliff; a near-zero gradient means a plateau.
Both follow from the geometry: the rate of change in an arbitrary direction is the dot product of the gradient with that direction, and a dot product is maximised when the two vectors align.
The immediate consequence for training: to decrease a loss, step in the direction of the negative gradient. That single sentence is gradient descent.
w ← w − η ∇L(w)
Scaling to millions of parameters
A neural network's loss is a function of every weight in it. A model with 10 million parameters has a loss surface in 10-million-dimensional space, and its gradient is a vector with 10 million entries — one partial derivative per weight, each saying "if I nudge this one weight, how does the loss respond?"
Nobody computes those one at a time. Backpropagation evaluates the whole gradient in roughly the cost of one forward pass, by applying the chain rule backwards through the network and reusing the shared factors. That efficiency is the reason deep learning works at all.
The mental picture that survives the jump in dimensions: you are standing somewhere on a landscape, you can feel which way is downhill, and you take a step. You cannot see the whole landscape and you do not need to.
Reading the gradient during training
Gradient values are one of the most informative diagnostics available while a model trains.
What you see
What it usually means
Gradient norm near zero early on
Dead neurons, saturated activations, or a bad initialisation
Gradient norm exploding to NaN
Learning rate too high, or no clipping in a recurrent model
Gradients fine at the output, tiny at the input
Vanishing gradients through depth
Loss falling but gradients steady
Healthy training
Loss flat and gradients large
Learning rate too high — bouncing across a valley
Logging the gradient norm per layer costs almost nothing and answers most "why is this not training" questions faster than staring at the loss curve.
Measure both partials
Each partial derivative holds the other variable still. Together they are the gradient, which points the steepest way up.
example_01.pyNumPy
import numpy as np
# f(x, y) = x^2 + 3xy + y^2
def f(x, y): return x ** 2 + 3 * x * y + y ** 2
x, y, h = 1.0, 2.0, 1e-6
df_dx = (f(x + h, y) - f(x, y)) / h # y held still
df_dy = (f(x, y + h) - f(x, y)) / h # x held still
print("f(x,y) = x^2 + 3xy + y^2 at (1, 2)")
print(" df/dx numerically %.5f by hand 2x+3y = %.1f" % (df_dx, 2 * x + 3 * y))
print(" df/dy numerically %.5f by hand 3x+2y = %.1f" % (df_dy, 3 * x + 2 * y))
print()
grad = np.array([2 * x + 3 * y, 3 * x + 2 * y])
print("gradient =", grad, " length %.3f" % np.linalg.norm(grad))
print()
print("stepping along the gradient increases f the fastest:")
for step in (0.0, 0.01, 0.02):
p = np.array([x, y]) + step * grad / np.linalg.norm(grad)
print(" step %.2f -> f = %.6f" % (step, f(p[0], p[1])))
Output
Experiments to try
Click around the bowl. The gradient always points away from the centre — uphill — and shrinks as you approach the minimum.
Press Run on the bowl. The path heads almost straight to the middle, decelerating as the gradient dies.
Switch to the valley (x² + 4y²) and run again. The path zig-zags, because the surface is far steeper in y than in x — the exact pathology that Adam and momentum were invented to fix.
Try the saddle and sit near the origin. One partial pushes up, the other down; the gradient nearly vanishes even though this is not a minimum — a genuine trap in high-dimensional optimisation.
Raise the learning rate above 0.4 on the valley and run. The steps overshoot and the point diverges — the classic symptom of too large a learning rate.
Summing up
Hold every other variable still to get a partial derivative; collect the partials to get the gradient. It points uphill, its length measures steepness, and stepping against it is what training a neural network literally means.
Directional derivatives, and the Hessian
Two related ideas complete the picture.
The directional derivative asks how fast f changes if you walk in some specific direction u (a unit vector). It is simply the dot product:
Duf = ∇f · u
Walk along the gradient and you get its full magnitude. Walk at right angles to it and you get zero — you are moving along a contour, staying at the same height. That is why contour lines and gradients always meet at right angles in the pictures.
The Hessian is the matrix of second partial derivatives, describing curvature. Its eigenvalues classify a stationary point: all positive means a minimum, all negative a maximum, mixed signs a saddle.
That last case matters in practice. In high dimensions, points where the gradient vanishes are overwhelmingly likely to be saddles rather than local minima — for all the eigenvalues to share a sign is improbable when there are millions of them. This is a large part of why training deep networks works better than early theory feared: the algorithm is escaping saddles, not getting trapped in bad minima.
Second-order methods use the Hessian to choose better steps, but storing an n×n matrix for n = 10⁶ is impossible, which is why first-order methods with clever momentum (Adam and its relatives) dominate in practice.
Questions people ask
What is the difference between a derivative and a gradient? A derivative is for one input; a gradient is the vector of partial derivatives for many.
Why the minus sign in gradient descent? The gradient points uphill, and you want to go down.
Does a zero gradient mean training is finished? It means you are at a stationary point. It could be a minimum, a saddle, or a plateau — and in deep learning the loss rarely reaches an exact zero gradient anyway.
What does the gradient's magnitude tell me? How steep the surface is. It is the number to log when diagnosing vanishing or exploding gradients.
Do I ever compute partials by hand? Rarely — autograd handles it. Understanding them is what makes learning rates, initialisation and architecture choices make sense.
What is a stochastic gradient? The gradient computed on a small batch rather than the whole dataset. It is a noisy estimate of the true gradient, and the noise turns out to help escape saddles.
Recap in one screen
A partial derivative measures the effect of one input with the others held fixed.
The gradient collects them into a vector that points in the direction of steepest increase.
Its length measures steepness; a near-zero gradient means a flat region.
Gradient descent steps against the gradient, scaled by the learning rate.
Backpropagation computes the whole gradient for the price of about one forward pass.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Start here”?
Standing on a hillside, "how steep is it?" has no single answer — it depends which way you face. A partial derivative answers it for one fixed direction; the gradient bundles those answers into a vector pointing straight up the slope.
What does this module say about “Partials: Freeze Everything Else”?
To compute ∂f/∂x you treat y as a constant and differentiate normally. That is the whole idea — you are asking how f changes if you step east while refusing to move north.
What does this module say about “Gradient Descent: Just Walk Backwards”?
Training a model means finding the lowest point of a loss surface. The gradient points uphill, so you step the other way:
Cheat sheet
Partial Derivatives and the Gradient
With two inputs there is no single slope — it depends which way you walk. Freeze one variable at a time to get the partials, then combine them into the gradient: the arrow pointing straight uphill.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.