Visualize how different optimization algorithms navigate the "Loss Landscape" to find the global minimum. Run multiple in parallel to observe realistic relative speeds!
Every optimiser here computes the same gradient. They differ entirely in how much they remember about the gradients that came before.
Picture a long narrow valley where the gradient across the valley is 10 and along it is 0.1. With η = 0.01, SGD steps 0.1 across and 0.001 along — a hundred to one. It bounces off the steep walls while creeping toward the actual minimum.
Momentum fixes this by accumulation: the across-valley components alternate sign and cancel, while the along-valley components all point the same way and add up. RMSprop fixes it differently, by normalising both directions to roughly equal step sizes.
Plain SGD steps in the direction of the current gradient and forgets everything before it. Momentum keeps a running average of past gradients and steps along that instead:
v ← βv + (1 − β)∇L w ← w − ηv
With the usual β = 0.9 the velocity is an average over roughly the last ten gradients. Consistent directions accumulate and speed up; directions that flip sign each step cancel out. That is exactly the fix for a narrow valley, where the gradient across the valley alternates while the gradient along it stays constant — the oscillation cancels and the useful direction survives.
Adam keeps two running averages per parameter: the mean of the gradients (momentum, as above) and the mean of their squares. It divides the step by the square root of that second average:
w ← w − η · m̂ / (√v̂ + ε)
The effect is a per-parameter learning rate. A parameter that has consistently seen large gradients gets a smaller effective step; one that has seen small or infrequent gradients gets a larger one. That is what makes Adam so effective on sparse data, where rare features would otherwise learn thousands of times more slowly than common ones.
The cost is memory: two extra values per parameter, so optimiser state is three times the size of the model. On a large model that is a real constraint, and it is why plain SGD with momentum is still used where memory is tight.
A practical note: the optimiser is not a substitute for the learning rate. Adam’s default of 0.001 is a reasonable starting point rather than a universal answer, and a badly chosen rate will defeat any optimiser on this list.
Plain gradient descent applies one learning rate to every parameter. That is a poor fit for a real network, where some parameters sit on steep slopes and others on nearly flat ones.
Optimisers are the successive fixes for that, and they build on each other:
| Optimiser | Idea added |
|---|---|
| SGD | Step against the gradient |
| SGD + momentum | Accumulate a velocity across steps |
| Nesterov | Look ahead before stepping |
| AdaGrad | Divide by accumulated squared gradients — per-parameter rates |
| RMSProp | Same, but with a decaying average so it does not stall |
| Adam | RMSProp plus momentum, with bias correction |
| AdamW | Adam with weight decay applied correctly |
Momentum is the first and biggest win. Averaging recent gradients builds speed in consistent directions and cancels oscillation across a narrow valley.
AdaGrad introduced per-parameter scaling, but its accumulator only grows, so the effective learning rate decays to zero and training stalls. RMSProp replaced the running sum with an exponential moving average, which fixed it.
Adam combines the two: a moving average of gradients (momentum) and of squared gradients (scaling), each bias-corrected for the first few steps when the averages start at zero.
m ← β₁m + (1−β₁)g v ← β₂v + (1−β₂)g²
w ← w − η · m̂ / (√v̂ + ε)
m is the smoothed gradient, v the smoothed squared gradient. Dividing by √v means a parameter with consistently large gradients takes smaller steps and one with tiny gradients takes larger ones — the per-parameter adaptation.
The defaults are unusually robust: β₁ = 0.9, β₂ = 0.999, ε = 1e-8. Those are almost never worth changing. β₂ = 0.999 corresponds to a memory of roughly a thousand steps, which is why Adam is stable but slow to react to a genuine change in the loss landscape.
AdamW is the version to use. Plain Adam applies weight decay by adding it to the gradient, which then gets divided by √v — so parameters with small gradients are penalised more, which was never the intent. AdamW applies the decay directly to the weights, decoupled from the adaptive scaling. The difference is measurable, and it is why every transformer recipe specifies AdamW.
Carrying a learning rate between optimisers. A good SGD learning rate is often 0.01–0.1; Adam's usual default is 0.001. Swapping optimiser while keeping η is one of the most common reasons a switch to Adam appears to make things worse.Assuming Adam always wins. It usually converges fastest, but well-tuned SGD with momentum frequently generalises better on vision tasks, and a good deal of published work still uses it for exactly that reason. Fast to a worse minimum is not a win.
Same downhill direction, different memory — and memory is what stops you bouncing off the walls.
| Situation | Optimiser | Starting learning rate |
|---|---|---|
| Transformers, language models | AdamW | 1e-4 to 3e-4, with warm-up |
| Vision, from scratch | SGD + momentum 0.9, or AdamW | 0.1 (SGD) / 1e-3 (AdamW) |
| Fine-tuning a pretrained model | AdamW | 1e-5 to 1e-4 |
| Tabular data, small networks | AdamW | 1e-3 |
| Sparse features, embeddings | Adam or Adagrad | 1e-3 |
The honest summary of a long-running debate: AdamW converges faster and needs less tuning; well-tuned SGD with momentum sometimes generalises slightly better on vision tasks. If you have limited time, use AdamW.
The schedule matters as much as the optimiser. Two components are standard:
Warm-up — ramp the learning rate from near zero over the first few hundred or few thousand steps. Adam's variance estimate is unreliable early on, and a large first step can damage the initialisation. Essential for transformers.
Cosine decay — smoothly reduce the rate to near zero over training. Reliably beats a constant rate, and beats step decay in most comparisons.
import torch
opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=3e-4, total_steps=epochs * len(loader), pct_start=0.1)
for batch in loader:
loss = criterion(model(batch.x), batch.y)
opt.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step()
sched.step() # per batch for OneCycle, not per epochAdam stores two extra values per parameter, so its optimiser state is twice the model size. For a 7-billion-parameter model in 32-bit that is 56GB of optimiser state alone, which is why large-model training uses 8-bit optimisers, sharding across devices (ZeRO), or memory-efficient variants such as Adafactor and Lion.
Lion keeps only momentum and uses the sign of the update, halving the state. Sophia and other second-order-flavoured methods use curvature estimates. These matter at scale; for ordinary work AdamW remains the right default.
A surface 1000x steeper in one direction than the other -- the normal situation in a real network. Plain SGD cannot solve it, and watching why explains every optimiser invented since.
Should I ever use plain SGD? With momentum, yes — it is still competitive on vision with a good schedule. Without momentum, essentially never.
Do I need to tune β₁ and β₂? Almost never. β₂ = 0.95 is sometimes used for very large models where the default is too slow to adapt.
Why does my loss spike after a while with Adam? Often the learning rate is too high for the later, sharper part of the landscape. A decaying schedule fixes it.
What weight decay should I use? 0.01 to 0.1 with AdamW is typical. Note that it applies to weights, and biases and normalisation parameters are usually excluded.
Is Adam bad for generalisation? The old claim is overstated. AdamW closed most of the gap, and with a proper schedule the difference is small.
Can I change optimiser mid-training? You can, and the state is lost, so expect a temporary disruption. Sometimes done deliberately: Adam early, SGD late.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
Same downhill direction, different memory — and memory is what stops you bouncing off the walls.
What does this module say about “What each one adds”?
Picture a long narrow valley where the gradient across the valley is 10 and along it is 0.1. With η = 0.01, SGD steps 0.1 across and 0.001 along — a hundred to one. It bounces off the steep walls while creeping toward the actual minimum.
What does this module say about “Why plain SGD zig-zags”?
Picture a long narrow valley where the gradient across the valley is 10 and along it is 0.1. With η = 0.01, SGD steps 0.1 across and 0.001 along — a hundred to one. It bounces off the steep walls while creeping toward the actual minimum.
Visualize how different optimization algorithms navigate the "Loss Landscape" to find the global minimum. Run multiple in parallel to observe realistic relative speeds!