Gradient descent batch processing
Visualize how different gradient descent strategies process data to update weights.
Overview
Three regimes
- Batch size 1 (stochastic). One sample per update. Very noisy gradient, very many updates, and the noise itself sometimes helps escape bad regions.
- Full batch. The whole dataset per update. The gradient is exact, but you get one update per epoch and it is slow.
- Mini-batch (32–256). What essentially everyone uses. Enough samples for a usable gradient estimate, enough updates to make progress, and it fits on a GPU.
Batch
Update @ Epoch End
Stochastic
Update Per Point
Mini-Batch
Update Per Batch
Gradient Descent Batch Processing: A Practical Guide
Batch size decides how many examples the model looks at before it changes its mind. It is the quietest hyperparameter with the widest effect on how training feels.
How the numbers work out
With 1,024 training samples:
- Batch size 32 → 32 updates per epoch
- Batch size 256 → 4 updates per epoch
Gradient noise falls roughly with the square root of batch size. Going from 32 to 128 — a 4× increase — halves the noise, not quarters it. That diminishing return is why batch sizes stop growing well before memory runs out.
Why mini-batch won
Batch gradient descent computes an exact gradient and needs a full pass over the data for a single update. On a million samples that is a million forward passes to move the weights once — accurate and unusable.
Stochastic gradient descent updates after every sample, so it makes a million updates per epoch, but each gradient is a single-sample estimate and the path jitters badly. It also wastes the hardware: one sample cannot occupy a GPU.
Mini-batches of 32 to 256 sit at the point where both problems disappear. The gradient estimate is close enough to the true one for stable progress, updates are frequent, and the batch is large enough to fill the device. This is not a compromise nobody is happy with — it is better than either extreme on both axes that matter.
Batch size and learning rate move together
These two cannot be tuned independently. The gradient of a batch is an average, so a larger batch gives a lower-variance estimate — and a more reliable estimate can support a larger step.
The linear scaling rule is the standard heuristic: multiply the batch size by k and multiply the learning rate by k. Going from batch 32 at learning rate 0.01 to batch 256 suggests roughly 0.08. Very large batches usually need a warmup period as well, ramping the rate up over the first few epochs, because the scaled rate is unstable while the weights are still random.
The corollary is that comparing two runs at different batch sizes with the same learning rate compares nothing useful.
Shuffling is not optional
Mini-batches must be drawn from a shuffled dataset, reshuffled every epoch. If the data arrives sorted by class — all the cats then all the dogs — then each batch contains one class, every gradient points toward predicting that class, and the model oscillates instead of converging.
Reshuffling each epoch also means the model never sees the same batch composition twice, which adds a small amount of useful noise. The exception is time series, where shuffling destroys the temporal ordering the model is supposed to learn from; there the batches must be contiguous windows.
Three sizes of step
The gradient is an average over training examples. How many you average over before updating defines three variants of the same algorithm.
| Variant | Examples per update | Gradient quality | Hardware use |
|---|---|---|---|
| Batch (full) | All of them | Exact | Poor — one update per pass |
| Stochastic | 1 | Very noisy | Poor — no parallelism |
| Mini-batch | 32–256 | Good estimate | Excellent |
Full-batch descent takes the true gradient and one step per pass through the data. On a million rows that is a million forward and backward passes for a single weight update, which is unusable.
Stochastic descent updates after every example. Steps are cheap and the gradient is a very noisy estimate; worse, a single 784-element vector uses a fraction of a percent of a GPU's capacity.
Mini-batch is the compromise everyone uses, and it wins on all three counts at once: a usable gradient estimate, a matrix multiplication large enough to saturate the hardware, and enough residual noise to escape saddle points.
Why the noise is useful
It is tempting to treat gradient noise as a defect to be minimised. It is not.
It escapes saddle points. In high dimensions almost every point with zero gradient is a saddle rather than a minimum. An exact gradient at a saddle is zero and full-batch descent stalls; a noisy one is not, and the model moves on.
It biases towards flat minima. Noise makes it hard to settle into a narrow crevice in the loss surface, and flat minima generalise better because a small shift in the data costs less. This is the main reason very large batches sometimes generalise slightly worse.
It acts as a regulariser. Part of why deep networks generalise better than their parameter counts suggest is this implicit regularisation from stochastic updates.
So the goal is not the cleanest possible gradient. It is a gradient good enough to point downhill, computed on a batch large enough to use the hardware.
Shuffling, and why it is not optional
Mini-batch descent assumes each batch is a representative sample. If the data arrives sorted — by class, by date, by source — then it is not.
Unshuffled sorted data produces batches containing a single class. The model spends one batch learning "everything is class A", the next unlearning it in favour of class B, and the gradients pull in contradictory directions. Batch normalisation makes it worse, since its statistics are computed per batch and become wildly unrepresentative.
Reshuffling every epoch matters too: it means the model never sees the same partition of examples twice, which adds useful variation.
loader = DataLoader(dataset, batch_size=64, shuffle=True, drop_last=True)drop_last=True discards a final partial batch. With batch normalisation that matters — a last batch of one or two examples produces meaningless statistics and a visibly worse update.
The exception is time series, where shuffling destroys the temporal ordering the model needs. There, batches are contiguous windows and the split is chronological.
Try this above
- Drag Batch Size to its minimum. The descent path visibly jitters — each step is reacting to one or two samples.
- Raise it toward the maximum. The path straightens out, but notice it also takes fewer, larger steps to cover the same ground.
- Now hold batch size high and lower Learning Rate. Progress stalls — a big smooth gradient with a tiny step size wastes the smoothness.
What usually goes wrong
Raising batch size without touching the learning rate. You get fewer updates per epoch, each no larger than before, so training slows down and people conclude the larger batch "trains worse". The usual remedy is the linear scaling rule: double the batch, double the learning rate, within reason.Comparing runs by epoch count. An epoch at batch size 512 contains a fraction of the updates an epoch at batch size 32 does. Compare by number of updates, or by wall-clock time, not by epochs.
In one line
Batch size trades gradient noise against how many times per epoch you get to move.
Choosing a batch size in practice
Start with the largest that fits comfortably in memory, up to about 256, then scale the learning rate to match.
The linear scaling rule is the standard guidance: double the batch size, double the learning rate. The reasoning is that a batch twice as large gives a gradient with roughly half the noise, so a step twice as long is equally safe. It holds well up to a few thousand examples per batch, with a warm-up period to survive the first large steps.
Beyond that, the noise that was providing implicit regularisation has largely gone, and specialised optimisers (LARS, LAMB) are needed to keep large-batch training competitive. This is why the papers that trained ImageNet in minutes needed more than just more GPUs.
Constraints that override the "as large as fits" rule:
- Batch normalisation needs at least about 8 examples per batch for usable statistics. Below that, use GroupNorm.
- Very large inputs — high-resolution images, long sequences — may force a batch of 1 or 2. Gradient accumulation recovers the effective batch size.
- Small datasets give few updates per epoch at a large batch size, so a smaller batch and more updates often trains better.
Gradient accumulation
When the batch you want does not fit, accumulate gradients over several small batches and update once:
accum = 8 # effective batch = 8 x batch_size
opt.zero_grad()
for i, batch in enumerate(loader):
loss = criterion(model(batch.x), batch.y) / accum # keep the average right
loss.backward() # accumulates into .grad
if (i + 1) % accum == 0:
opt.step()
opt.zero_grad()Dividing the loss by accum is essential — without it the accumulated gradient is eight times too large and the effective learning rate is eight times what you set.
This is the same accumulation behaviour that makes a forgotten zero_grad() a bug. Here it is the feature.
The cost is wall-clock time: eight small batches take about as long as one large one would have. You get the large-batch statistics, not the large-batch speed.
Full batch, one row, and everything between
The same problem solved by batch, stochastic and mini-batch gradient descent, with the noise each one injects measured directly -- because that noise is the whole difference.
Questions people ask
How many updates does one epoch give? dataset size / batch size. With 50,000 examples and a batch of 64, that is 782 — not 1.
Why did accuracy drop when I raised the batch size? Fewer updates per epoch and an unscaled learning rate. Scale the rate up and add warm-up.
Is a power of two required? No, though it aligns well with hardware and is marginally faster.
Should I use full-batch descent on a small dataset? Rarely — you lose the noise that helps generalisation, and get very few updates. Even on 1,000 rows, mini-batches usually train better.
Does batch size affect the final accuracy? Somewhat. Very large batches tend to generalise slightly worse; very small ones are slow and noisy.
What if my dataset does not divide evenly? The last batch is smaller. Drop it if batch normalisation is in use.
Recap in one screen
- Full batch is accurate and unusable; single-example updates are noisy and waste hardware; mini-batches win on both.
- The residual noise is useful — it escapes saddles and favours flat minima.
- Shuffle every epoch, and drop the final partial batch when using batch normalisation.
- Scale the learning rate linearly with batch size, with warm-up.
- Gradient accumulation buys a large effective batch on small hardware, at the cost of time.
Recall check
0 of 3Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What is meant by “Batch normalisation” here?
needs at least about 8 examples per batch for usable statistics. Below that, use GroupNorm.
What is meant by “Very large inputs” here?
— high-resolution images, long sequences — may force a batch of 1 or 2. Gradient accumulation recovers the effective batch size.
What is meant by “Small datasets” here?
give few updates per epoch at a large batch size, so a smaller batch and more updates often trains better.
Gradient Descent Batch Processing
Batch size decides how many examples the model looks at before it changes its mind. It is the quietest hyperparameter with the widest effect on how training feels.