Visualize how different optimization algorithms navigate the "Loss Landscape" to find the global minimum. Run multiple in parallel to observe realistic relative speeds!
The progression from SGD to Adam is a progression in memory: SGD remembers nothing, momentum remembers an exponential average of past gradients, RMSprop remembers an exponential average of past squared gradients. And Adam remembers both. Each addition solves a set failure mode — oscillation in ravines, slow progress on plateaus. And uneven param scales — at the cost of two extra state vectors per param. Adam with its default hyperparameters (lr=0.001, beta1=0.9, beta2=0.999) is the standard starting point for nearly all deep learning tasks. AdamW (Adam with decoupled weight decay) is the variant used in transformer training. For convex problems or when the dataset is very large, plain SGD with momentum and a well-tuned learning rate schedule can match or beat Adam. It is why it stays standard for training large pic classifiers. This page runs all four optimisers on the same loss surface, plots their trajectories. And shows the per-step state (velocity, squared-gradient average) that explains each path.
Visualize how different optimization algorithms navigate the "Loss Landscape" to find the global minimum.
Every optimiser here computes the same gradient. They differ entirely in how much they remember about the gradients that came before.
The progression from SGD to Adam is a progression in memory: SGD remembers nothing, momentum remembers an exponential average of past gradients, RMSprop remembers an exponential average of past squared gradients. And Adam remembers both. Each addition solves a set failure mode — oscillation in ravines, slow progress on plateaus. And uneven param scales — at the cost of two extra state vectors per param. Adam with its default hyperparameters (lr=0.001, beta1=0.9, beta2=0.999) is the standard starting point for nearly all deep learning tasks. AdamW (Adam with decoupled weight decay) is the variant used in transformer training. For convex problems or when the dataset is very large, plain SGD with momentum and a well-tuned learning rate schedule can match or beat Adam. It is why it stays standard for training large pic classifiers. This page runs all four optimisers on the same loss surface, plots their trajectories. And shows the per-step state (velocity, squared-gradient average) that explains each path.
Picture a long narrow valley where the gradient across the valley is 10 and along it is 0.1. With η = 0.01, SGD steps 0.1 across and 0.001 along — a hundred to one. It bounces off the steep walls while creeping toward the actual min.
Momentum fixes this by accumulation: the across-valley parts alternate sign and cancel, while the along-valley parts all point the same way and add up. RMSprop fixes it differently, by normalising both ways to roughly equal step sizes.
Plain SGD steps in the way of the current gradient and forgets all before it. Momentum keeps a running average of past gradients and steps along that instead:
v ← βv + (1 − β)∇L w ← w − ηv
With the usual β = 0.9 the velocity is an average over roughly the last ten gradients. Consistent ways accumulate and speed up. Ways that flip sign each step cancel out. That is just the fix for a narrow valley. There the gradient across the valley alternates while the gradient along it stays constant — the oscillation cancels and the useful way survives.
Adam keeps two running averages per param: the mean of the gradients (momentum, as above) and the mean of their squares. It divides the step by the square root of that second average:
w ← w − η · m̂. / (√v̂ + ε)
The effect is a per-param learning rate. A param that has always seen large gradients gets a less effective step. One that has seen small or infrequent gradients gets a more one. That is what makes Adam so effective on sparse data. There rare features would otherwise learn thousands of times more slowly than common ones.
The cost is memory: two extra vals per param. So optimiser state is three times the size of the net. On a large net that is a real constraint. And it is why plain SGD with momentum is still used where memory is tight.
A real note: the optimiser is not a substitute for the learning rate. Adam’s default of 0.001 is a reasonable starting point rather than a universal answer. And a badly chosen rate will defeat any optimiser on this list.
Plain gradient descent applies one learning rate to each param. That is a poor fit for a real network. There some params sit on steep slopes and others on nearly flat ones.
Optimisers are the successive fixes for that, and they build on each other:
| Optimiser | Idea added |
|---|---|
| SGD | Step against the gradient |
| SGD + momentum | Accumulate a velocity across steps |
| Nesterov | Look ahead before stepping |
| AdaGrad | Divide by accumulated squared gradients — per-parameter rates |
| RMSProp | Same, but with a decaying average so it does not stall |
| Adam | RMSProp plus momentum, with bias correction |
| AdamW | Adam with weight decay applied correctly |
Momentum is the first and biggest win. Averaging recent gradients builds speed in consistent ways and cancels oscillation across a narrow valley.
AdaGrad added per-param scaling, but its accumulator just grows. So the effective learning rate decays to zero and training stalls. RMSProp replaced the running sum with an exponential moving average, which fixed it.
Adam mixes the two: a moving average of gradients (momentum) and of squared gradients (scaling), each bias-corrected for the first few steps when the averages start at zero.
m ← β₁m + (1−β₁)g v ← β₂v + (1−β₂)g²
w ← w − η · m̂. / (√v̂ + ε)
m is the smoothed gradient, v the smoothed squared gradient. Dividing by √v means a param with always large gradients takes less steps and one with tiny gradients takes more ones — the per-param adaptation.
The defaults are unusually reliable: β₁ = 0.9, β₂ = 0.999, ε = 1e-8. Those are almost never worth changing. β₂ = 0.999 corresponds to a memory of roughly a thousand steps. It is why Adam is stable but slow to react to a genuine change in the loss landscape.
AdamW is the version to use. Plain Adam applies weight decay by adding it to the gradient. It then gets divided by √v — so params with small gradients are penalised more. It was never the intent. AdamW applies the decay right to the weights, decoupled from the adaptive scaling. The gap is measurable. And it is why each transformer recipe names AdamW.
Carrying a learning rate between optimisers. A good SGD learning rate is often 0.01–0.1. Adam's usual default is 0.001. Swapping optimiser while keeping η is one of the most common reasons a switch to Adam appears to make things worse.Assuming Adam always wins. It often converges fastest. But well-tuned SGD with momentum often generalises better on vision tasks. And a good deal of published work still uses it for just that reason. Fast to a worse min is not a win.
Same downhill way, other memory — and memory is what stops you bouncing off the walls.
| Situation | Optimiser | Starting learning rate |
|---|---|---|
| Transformers, language models | AdamW | 1e-4 to 3e-4, with warm-up |
| Vision, from scratch | SGD + momentum 0.9, or AdamW | 0.1 (SGD) / 1e-3 (AdamW) |
| Fine-tuning a pretrained model | AdamW | 1e-5 to 1e-4 |
| Tabular data, small networks | AdamW | 1e-3 |
| Sparse features, embeddings | Adam or Adagrad | 1e-3 |
The honest summary of a long-running debate: AdamW converges faster and needs less tuning. Well-tuned SGD with momentum sometimes generalises slightly better on vision tasks. If you have limited time, use AdamW.
The schedule matters as much as the optimiser. Two parts are standard:
Warm-up — ramp the learning rate from near zero over the first few hundred or few thousand steps. Adam's variance estimate is unreliable early on. And a large first step can damage the initialisation. Key for transformers.
Cosine decay — smoothly reduce the rate to near zero over training. Reliably beats a constant rate, and beats step decay in most comparisons.
import torch
opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=3e-4, total_steps=epochs * len(loader), pct_start=0.1)
for batch in loader:
loss = criterion(model(batch.x), batch.y)
opt.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step()
sched.step() # per batch for OneCycle, not per epochAdam stores two extra vals per param. So its optimiser state is twice the net size. For a 7-billion-param net in 32-bit that is 56GB of optimiser state alone. It is why large-net training uses 8-bit optimisers, sharding across devices (ZeRO). Or memory-fast variants such as Adafactor and Lion.
Lion keeps just momentum and uses the sign of the update, halving the state. Sophia and other second-order-flavoured ways use curvature estimates. These matter at scale. For ordinary work AdamW stays the right default.
A surface 1000x steeper in one way than the other -- the normal case in a real network. Plain SGD cannot solve it. And watching why explains each optimiser invented since.
Building on the same setup, the next block shows the left cases.
The next block covers the left steps.
The next block continues with the left demonstrations.
Building on the same setup, the next block shows the left cases.
Should I ever use plain SGD? With momentum, yes — it is still strong on vision with a good schedule. Without momentum, in effect never.
Do I need to tune β₁ and β₂? Almost never. β₂ = 0.95 is sometimes used for very large nets where the default is too slow to adapt.
Why does my loss spike after a while with Adam? Often the learning rate is too high for the later, sharper part of the landscape. A decaying schedule fixes it.
What weight decay should I use? 0.01 to 0.1 with AdamW is typical. Note that it applies to weights. And biases and normalisation params are often excluded.
Is Adam bad for generalisation? The old claim is overstated. AdamW closed most of the gap. And with a proper schedule the gap is small.
Can I change optimiser mid-training? You can. And the state is lost, so expect a temporary disruption. Sometimes done deliberately: Adam early, SGD late.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “What each one adds”?
What does this module say about “Why plain SGD zig-zags”?
Run several optimizers in parallel on one loss surface and see how momentum and adaptive learning rates change the route.