Understand how a model learns parameters by minimizing loss over iterations.
Overview
The one rule the whole algorithm follows
You have a loss function that says how wrong the model currently is, and parameters you can change. Gradient descent computes the derivative of the loss with respect to each parameter — the direction in which loss increases fastest — and then moves the opposite way:
w ← w − η · ∂L/∂w
η is the learning rate: how far to step. The gradient supplies the direction; the learning rate supplies the distance. That is the entire algorithm, repeated until the loss stops falling.
The reason it works at all is that the gradient is a local answer — it needs no knowledge of the loss surface beyond the current point. That is what makes it usable on a function with a hundred million parameters that nobody can visualise.
REGRESSION PLOTDrag/Click to nudge weights
Batch
Stochastic
Mini-Batch
LOSS HISTORY
Gradient Descent: A Practical Guide
Follow the slope downhill, one step at a time. Every parameter in every neural network is found this way, and the step size decides whether it works at all.
Work one step through by hand
Fit y = wx to the single point (2, 6), starting at w = 1, with squared error loss:
The new prediction is 5.2, much closer to 6. Note the gradient was negative, so subtracting it increased w — the minus sign in the update rule is what turns “direction of steepest increase” into “step downhill”.
Batch, stochastic, and mini-batch
The gradient is a sum over training examples, and how many you include before each update defines the three variants:
Batch — use every example. The gradient is exact, so the path to the minimum is smooth, but one update requires a full pass over the data.
Stochastic (SGD) — use one example. Updates are extremely cheap and extremely noisy; the path jitters, which costs precision but can jolt the model out of poor regions.
Mini-batch — use 32 to 256 examples. Close to the exact gradient, cheap enough to update often, and the shape that matches how GPUs actually work. This is what essentially all real training uses.
Walking downhill on the loss surface
Training is a search. There is a loss function that scores how wrong the model is, and gradient descent is the procedure for finding parameter values that make it small.
w ← w − η × ∂L/∂w
The gradient points uphill, so the minus sign walks downhill. η is the learning rate: how far to step.
Worked through on one parameter. Loss L = w², so the gradient is 2w. Start at w = 4 with η = 0.1:
Step
w
Gradient
New w
1
4.00
8.00
3.20
2
3.20
6.40
2.56
3
2.56
5.12
2.05
10
0.43
0.86
0.34
Two things are visible. It converges towards zero, the true minimum. And the steps shrink as it approaches, because the gradient shrinks — automatic slowing down, with no schedule required.
The learning rate decides everything
Learning rate
Behaviour
Far too high
Loss oscillates or explodes to NaN
Slightly too high
Loss falls then plateaus high, bouncing across the valley
About right
Loss falls steadily and flattens smoothly
Too low
Loss falls, very slowly, and may never arrive
There is a cheap way to find it rather than guessing: run a short learning-rate sweep, increasing the rate exponentially over a few hundred batches while recording the loss. The loss falls, reaches a minimum, then explodes. Pick a value somewhat below where it exploded — often the steepest part of the descent.
This takes a minute or two and is worth more than any other single piece of tuning.
Full batch, stochastic, and mini-batch
The gradient is really an average over training examples, and how many you average over is a design choice.
Full batch uses the entire dataset for every update. The gradient is accurate and each step is expensive; on a million rows it is unusable.
Stochastic (one example) updates after every single row. Steps are cheap and extremely noisy, and it cannot exploit vectorised hardware.
Mini-batch uses 32 to 256 examples per update, and it is what everyone actually uses. The gradient is a reasonable estimate, the batch fills a GPU efficiently, and the remaining noise is useful — it helps the optimiser escape saddle points and shallow local minima.
An epoch is one pass through the data; with 50,000 rows and a batch size of 64 that is 782 updates per epoch, not one.
Batch size and learning rate move together: doubling the batch size roughly halves the gradient noise, and the usual rule is to scale the learning rate linearly with batch size (with a short warm-up for very large batches).
The loop, and what the learning rate does to it
Gradient descent is three lines repeated. This runs them on a problem with a known answer, then sweeps the learning rate through working, crawling, and diverging.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
n = 400
X = np.column_stack([np.ones(n), rng.normal(size=(n, 2))])
true_w = np.array([4.0, 2.5, -1.5])
y = X @ true_w + rng.normal(0, 0.5, n)
def descend(lr, steps=200, record=()):
w = np.zeros(3)
hist = []
for s in range(steps):
pred = X @ w
grad = 2 * X.T @ (pred - y) / n # d/dw of mean squared error
w = w - lr * grad
if s in record:
hist.append((s, ((X @ w - y) ** 2).mean(), w.copy()))
return w, ((X @ w - y) ** 2).mean(), hist
print("the loop, in full:")
print(" pred = X @ w")
print(" grad = 2 * X.T @ (pred - y) / n")
print(" w = w - lr * grad")
print()
w, mse, hist = descend(0.1, record=(0, 1, 2, 5, 20, 50, 199))
print("lr = 0.1, watching it arrive:")
print("%8s %12s %s" % ("step", "MSE", "weights"))
for s, m, ww in hist:
print("%8d %12.6f %s" % (s, m, np.round(ww, 4)))
print(" true weights:", true_w)
print()
print("the closed form agrees, because this problem has one:")
exact = np.linalg.solve(X.T @ X, X.T @ y)
print(" gradient descent:", np.round(w, 6))
print(" normal equation :", np.round(exact, 6))
print(" (a neural network has no closed form. that is the only reason we")
print(" iterate at all.)")
print()
print("now the learning rate, which is the whole game:")
print("%10s %14s %16s %s" % ("lr", "MSE @ 200", "distance to w*", "verdict"))
for lr in (0.001, 0.01, 0.1, 0.5, 0.9, 1.05):
w2, m2, _ = descend(lr)
d = np.linalg.norm(w2 - exact)
if not np.isfinite(m2) or m2 > 1e6:
verdict = "DIVERGED"
elif d > 1.0:
verdict = "too slow, still travelling"
elif d > 1e-4:
verdict = "nearly there"
else:
verdict = "converged"
print("%10.3f %14s %16s %s"
% (lr, "%.6f" % m2 if np.isfinite(m2) and m2 < 1e9 else "overflow",
"%.6f" % d if np.isfinite(d) and d < 1e9 else "-", verdict))
print()
print("too small and 200 steps are not enough to arrive. too large and each")
print("step overshoots further than the last, so the loss grows without")
print("bound. there is a real threshold here, not a matter of taste:")
eig = np.linalg.eigvalsh(2 * X.T @ X / n).max()
print(" divergence starts above 2/L, where L is the largest curvature.")
print(" L = %.4f, so the limit is %.4f." % (eig, 2 / eig))
print()
print("the gradient shrinks as you approach the answer, so the steps do too:")
w3 = np.zeros(3)
print("%8s %14s %14s" % ("step", "|gradient|", "step size"))
for s in range(6):
g = 2 * X.T @ (X @ w3 - y) / n
print("%8d %14.6f %14.6f" % (s, np.linalg.norm(g), 0.1 * np.linalg.norm(g)))
w3 = w3 - 0.1 * g
print(" that is automatic braking. it is also why a fixed learning rate")
print(" works at all, and why schedulers exist for when it does not.")
Output
Try it yourself
Find the working range. Set Learning Rate to 0.01 and press Train. The line converges smoothly onto the data. This is what a healthy run looks like.
Make it too small. Set Learning Rate to 0.001 and press Reset, then Train. The line still moves in the right direction but crawls — correct, and far too slow to be useful.
Make it too large. Set Learning Rate to 0.1 and train again. The fit overshoots the minimum and the loss oscillates or diverges outright. Steps too big do not just slow convergence, they prevent it.
Compare the three methods. Press Run All 3 and watch the paths together. Batch is smooth, stochastic is jagged, mini-batch sits between — and all three end up in roughly the same place.
Add noise. Raise Noise and press Data, then train. The loss no longer approaches zero; it flattens out above it. That floor is irreducible error, not a training failure.
What trips people up
Learning rate too high. Loss goes to NaN or oscillates upward. This is the first thing to check when training diverges, and dividing it by 10 is the first thing to try.
Learning rate too low. Loss falls, painfully slowly, and can stall on a plateau long enough to look converged when it is not.
Unscaled features. A feature in the tens of thousands and one in decimals produce wildly different gradient magnitudes, so no single learning rate suits both. Scale inputs before training, not after diagnosing.
Reading one noisy step as a trend. With mini-batches the loss goes up on individual steps all the time. Judge the moving average, not the last value.
What to remember
Gradient descent repeatedly moves each parameter against its gradient, scaled by a learning rate, and that single rule trains every neural network in use. The gradient is reliable; the learning rate is the choice you have to get right, because too large diverges and too small never arrives. Mini-batches are the standard compromise between the exact-but-expensive batch gradient and the cheap-but-noisy stochastic one.
Momentum, and why plain descent is rarely used
Plain gradient descent is inefficient in a common situation: a loss surface that is much steeper in one direction than another. The steps zigzag across the narrow direction and crawl along the shallow one.
Momentum fixes it by accumulating a velocity:
v ← βv + ∂L/∂w w ← w − ηv
With β = 0.9, the update is an exponentially weighted average of recent gradients. Consistent directions build up speed; oscillating ones partly cancel. It is the single cheapest improvement over plain descent, and effectively nobody trains without it or something like it.
Adam goes further by keeping a per-parameter estimate of both the mean and the variance of recent gradients, and dividing by the square root of the variance. Parameters with consistently small gradients get larger effective steps. This is why Adam works well with almost no tuning, and why AdamW — the variant that decouples weight decay from the adaptive scaling — is the default for transformers.
Plain SGD with momentum still wins on some vision benchmarks, generalising slightly better with a well-tuned schedule. AdamW is the safer starting point.
What a healthy training run looks like
Loss falls quickly at first, then more slowly, then flattens.
Validation loss follows training loss down, then flattens or turns up. The turn is where overfitting begins and where early stopping should trigger.
Gradient norms stay in a similar range across layers and across time.
The loss is noisy batch to batch and smooth when averaged over an epoch. Batch-level noise is expected, not a problem.
And the failures, with their usual causes:
Symptom
Likely cause
Loss is NaN
Learning rate too high, or log(0) in the loss
Loss flat from step one
Learning rate far too low, or a bug in the data
Loss falls then jumps up permanently
Learning rate too high for the later, sharper region
Training loss falls, validation rises
Overfitting — regularise or stop earlier
Both losses stuck high
Underfitting — bigger model, better features, longer training
Questions people ask
Which optimiser should I use? AdamW at 1e-3 (or 3e-4 for transformers) as a default. Try SGD with momentum and a cosine schedule if you are chasing the last fraction on a vision benchmark.
Does gradient descent find the global minimum? Almost never, and it does not matter. In high dimensions there are vast numbers of near-equivalent good minima, and the noise from mini-batches helps escape the bad regions.
Why is my loss noisy? Because each batch is a different sample. Judge progress on epoch averages, not single batches.
Should the learning rate change during training? Yes — a warm-up followed by cosine decay is the modern default and reliably beats a constant rate.
How do I know when to stop? When validation loss stops improving for a set number of epochs. Keep the best checkpoint, not the last.
What is gradient accumulation? Calling backward() several times before step(), so a small GPU can simulate a large batch.
Recap in one screen
Step against the gradient, scaled by the learning rate; repeat per batch.
The learning rate matters more than anything else — find it with a short sweep.
Mini-batches of 32–256 balance gradient quality, hardware efficiency and useful noise.
Momentum and Adam accelerate progress and damp oscillation; plain descent is rarely used.
Watch training and validation loss together, and stop when validation stops improving.
Check yourself
0 of 3
Answer without scrolling back up.
Your loss oscillates wildly and sometimes increases. The most likely cause is:
Steps large enough to overshoot the minimum bounce from one wall of the valley to the other. Cutting the learning rate is the first thing to try.
The loss decreases, but almost imperceptibly, over thousands of steps. This suggests:
Tiny steps make real but glacial progress. It is the mirror image of the previous failure, and the reason schedules start high and decay rather than picking one value forever.
Why subtract the gradient instead of adding it?
The gradient is the direction of steepest increase. Moving against it is what makes the algorithm gradient *descent*.
Cheat sheet
Gradient Descent
You have a loss function that says how wrong the model currently is, and parameters you can change. Gradient descent computes the derivative of the loss with respect to each parameter — the direction in which loss increases fastest — and then moves the opposite way:
DEEP LEARNING · vizlearn.in/deep_learning/gradient_descent_training.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.