Adjust Complexity and Epochs to watch Polynomial Regression overfit the training data. See how Scheduled Learning Rates succeed where Fixed Learning Rates bounce and fail.
Early in training you are far from any good solution and want large steps. Late in training you are close, and large steps make you bounce around the minimum without ever landing in it. A fixed rate forces a compromise that is wrong at both ends.
The common schedules:
A single fixed learning rate has to be two things at once: big enough to cross the landscape early, small enough to settle at the end. A schedule lets it be both in turn.
Start at η0 = 0.1 and multiply by 0.1 every 30 epochs:
The characteristic sign that it is working is a visible drop in loss immediately after each decay step: the model had been oscillating around a minimum it could not land in, and a smaller step lets it descend the rest of the way.
Early in training the weights are far from anything useful, and large steps make rapid progress. Late in training the model is close to a good solution, and large steps overshoot it repeatedly — the loss plateaus and bounces rather than settling.
A schedule changes the learning rate over the course of training to suit both phases: large early, small late.
The effect is not marginal. A well-scheduled run typically reaches a noticeably lower final loss than the best constant rate, and the schedule is one of the cheapest improvements available.
| Schedule | Shape |
|---|---|
| Constant | Flat throughout |
| Step decay | Drops by a factor at fixed epochs |
| Exponential | Multiplied by a constant each epoch |
| Cosine | Smooth decay following half a cosine curve |
| One-cycle | Warm-up, then cosine decay to near zero |
| Reduce-on-plateau | Drops when validation stops improving |
Starting at full learning rate is often the wrong thing to do, for two specific reasons.
Adam's variance estimate is built from an exponential moving average that starts at zero, so for the first few hundred steps it is unreliable and the effective step size can be far larger than intended.
And a randomly initialised model produces meaningless gradients; taking a large step along them damages the initialisation more than it helps.
Warm-up ramps the rate linearly from near zero over the first few hundred to few thousand steps. It is effectively mandatory for transformers, and it is a large part of why very large models train stably at all.
Combining warm-up with cosine decay gives the shape almost every modern recipe uses: up quickly, then a long smooth descent to near zero.
sched = torch.optim.lr_scheduler.OneCycleLR(
opt, max_lr=3e-4,
total_steps=epochs * len(loader),
pct_start=0.1, # 10% of the run spent warming up
)
for batch in loader:
...
opt.step()
sched.step() # per batch for OneCycle, not per epochCalling sched.step() once per epoch when the scheduler expects per-batch stepping is a common bug: the schedule finishes hundreds of times too slowly and the run behaves as if the rate were constant.
The schedule needs a peak value, and there is a systematic way to find it rather than guessing.
Run a short learning-rate sweep: increase the rate exponentially from about 1e-7 to 1 over a few hundred batches, recording the loss. The curve falls, reaches a minimum, and then explodes upwards. Pick a value somewhat below the explosion point — often around the steepest descent.
That takes a minute or two of compute and is worth more than any other single act of tuning. The learning rate affects results more than the optimiser choice, the layer count or the width.
Decaying too early. Shrink the step before the model has reached a good region and it will crawl the rest of the way, converging neatly to somewhere mediocre.Scheduling on epochs when the real unit is updates. Change the batch size and an epoch contains a different number of steps, so a schedule tuned at one batch size silently becomes a different schedule at another.Tuning the schedule before the initial rate. The starting value matters more than the decay shape. Find a good η0 first, then decide how it should come down.
Big steps to travel, small steps to arrive — and get the starting size right before tuning the decay.
| Situation | Schedule |
|---|---|
| Transformers, language models | Linear warm-up then cosine decay |
| Vision, training from scratch | One-cycle, or cosine with warm-up |
| Fine-tuning | Low constant rate, or a short cosine decay |
| Unknown training length | Reduce-on-plateau |
| Reproducing a paper | Whatever it specifies — usually step decay |
Cosine decay has become the default because it beats step decay in most comparisons and has one fewer thing to tune — no decision about where the drops go.
Reduce-on-plateau is the pragmatic choice when you do not know how long training will take: it watches validation loss and cuts the rate by a factor (typically 10) when improvement stalls. It requires no total-steps estimate, which the cosine schedules do.
Warm restarts (SGDR) repeatedly decay to near zero and jump back up. The jumps can knock the model out of a sharp minimum into a better basin, and each cycle's endpoint makes a useful ensemble member.
Failures and their signatures:
| Symptom | Likely cause |
|---|---|
| Loss spikes early | Warm-up too short, or peak rate too high |
| Loss plateaus and bounces | Rate too high for the current phase |
| Loss falls but never flattens | Training ended before the schedule did |
| Loss barely moves at all | Peak rate too low, or scheduler stepped per epoch instead of per batch |
| Sudden improvement at the very end | Normal — the final low-rate phase settling into a minimum |
That last row is worth expecting: a visible drop in the closing epochs is the schedule working as intended, not a fluke.
A fixed learning rate cannot be both fast at the start and precise at the end. Each schedule here is plotted as numbers and then run on the same problem.
Do I need a schedule with Adam? Yes. Adam adapts per-parameter scaling; it does not replace a global decay, and every large-model recipe uses both.
How long should warm-up be? 5–10% of total steps, or a fixed few thousand steps for large models.
Should I step the scheduler per batch or per epoch? It depends on the scheduler. OneCycle and cosine-with-total-steps expect per batch; StepLR and ReduceLROnPlateau expect per epoch. Read the documentation rather than guessing.
What if I do not know the total number of steps? Use reduce-on-plateau, which needs no estimate.
Can the rate go to exactly zero? Cosine schedules usually decay to a small floor rather than zero, since zero means no further learning at all.
Does the schedule interact with batch size? Yes — larger batches generally want a proportionally larger peak rate, with a longer warm-up to match.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
Big steps to travel, small steps to arrive — and get the starting size right before tuning the decay.
What does this module say about “Why one value cannot serve both phases”?
Early in training you are far from any good solution and want large steps. Late in training you are close, and large steps make you bounce around the minimum without ever landing in it. A fixed rate forces a compromise that is wrong at both ends.
What does this module say about “One rate does not fit the whole run”?
Early in training the weights are far from anything useful, and large steps make rapid progress. Late in training the model is close to a good solution, and large steps overshoot it repeatedly — the loss plateaus and bounces rather than settling.
Early in training you are far from any good solution and want large steps. Late in training you are close, and large steps make you bounce around the minimum without ever landing in it. A fixed rate forces a compromise that is wrong at both ends.