vizlearn
Home / Deep Learning

Batch Processing in the Neural Network

Deep LearningBy Updated

Interactive architecture builder. Drag to pan, scroll to zoom, right-click to edit.

Data Rows 100
0%

Analysis

Layers -
Neurons -
Total Params -

Selection

Hover over nodes for details.
Right-click to edit | Scroll to Zoom

Overview

Interactive architecture builder.

In this guide
  1. A batch is a matrix, not a loop
  2. Epoch, batch, iteration
  3. Why cases are handled in groups
  4. Epochs, steps and the counts that confuse people
  5. Batch size, learning rate and generalisation
  6. Why we process many rows at once
  7. Things to try
  8. How batch size affects the result, not just the speed
  9. Where this goes wrong
  10. Key takeaway
  11. Gradient accumulation: a big batch on a small GPU
  12. What limits batch size
  13. Batching at inference
  14. Questions people ask
  15. Recap in one screen

Batch Processing in Networks: A Practical Guide

Networks process many samples at once, not one at a time. Batching is what makes GPUs worth using - and the batch size quietly changes both how fast you train and how well the result generalises.

A batch is a matrix, not a loop

Feeding one sample through a layer is a vector-matrix product. Feeding 32 samples is a matrix-matrix product — stack the 32 input vectors into a 32×nin matrix and multiply once by the same nin×nout weights.

Crucially the weights are unchanged. Batching does not alter the net. It changes how many samples pass through it per step. Since the hardware executes one large matrix multiply far more well than 32 small ones, a batch of 32 costs nothing like 32 times a single sample. The gradient computed on a batch is the average of the per-sample gradients. So more batches give a more exact estimate of the true gradient but update the weights fewer times per epoch. Small batches (8-64) add noise that acts as implicit regularisation, often improving generalisation. Large batches (256-4096) are more stable but can converge to sharper minima that generalise worse. The batch size also finds GPU memory usage: each sample in the batch needs its own activations stored for backpropagation. So doubling the batch size roughly doubles the memory. Gradient accumulation simulates a large batch by summing gradients across a few small forward passes before updating — same effective batch size, less memory. The relationship between batch size and learning rate matters: when you scale the batch size up by k, scaling the learning rate up by k (linear scaling rule) keeps training dynamics close, up to a point. This page runs the same net with other batch sizes, shows the loss curve and gradient variance for each. And covers the memory-accuracy tradeoff.

Epoch, batch, iteration

These three get muddled constantly:

  • Batch size — samples handled before one weight update.
  • Iteration (step) — one forward pass, one backward pass, one update.
  • Epoch — one complete pass over the training set.

With 10,000 samples and a batch size of 100, one epoch is 100 iterations. So 10 epochs means 1,000 weight updates. Halving the batch size to 50 doubles the updates per epoch to 200 — which is why batch size and learning rate cannot be tuned independently.

Why cases are handled in groups

Training could update the weights after each single case. Or after seeing the whole dataset. Both are worse than the middle option, and for other reasons.

One case at a time gives an very noisy gradient estimate and, more importantly, wastes the hardware. A GPU has thousands of cores designed to do the same step on many vals at once. Feeding it one 784-item vector uses a fraction of a percent of that capacity.

The whole dataset gives an exact gradient at enormous cost per update. On a million rows, a single step needs a million forward and backward passes. And the net gets one weight update for all that work.

A mini-batch of 32 to 256 is what everyone uses. The gradient is a reasonable estimate, the matrix multiplications are large enough to saturate the hardware. And the left noise is useful — it helps the optimiser escape saddle points.

The arithmetic is a matrix multiply either way: a batch of 64 cases with 784 features is a 64×784 matrix multiplied by the layer's weights. It is one step rather than 64.

Epochs, steps and the counts that confuse people

TermMeaning
Batch sizeExamples per update
Step / iterationOne forward pass, one backward pass, one update
EpochOne full pass through the training data
Steps per epochdataset size / batch size

With 50,000 training cases and a batch size of 64, one epoch is 782 steps — 782 weight updates, not one. Ten epochs is 7,820 updates.

That relationship has a consequence people trip over: doubling the batch size halves the count of updates per epoch. So a run with batch size 256 and the same count of epochs does a quarter of the updates of one with batch size 64. And will often be behind unless the learning rate is scaled up to compensate.

Batch size, learning rate and generalisation

More batches give less noisy gradients, so more steps are safe. The standard rule is to scale the learning rate linearly with batch size — double the batch, double the rate — with a warm-up period, since the first few large steps are the risky ones.

That rule holds well up to a point. Beyond a few thousand cases per batch, the gradient noise turns so small that the implicit regularisation it provided disappears. And very large-batch training tends to generalise slightly worse. Techniques such as LARS and LAMB exist namely to push that ceiling more.

Batch sizeCharacter
1–8Very noisy; slow on GPUs; sometimes necessary for huge inputs
32–256The normal range; good hardware use, useful noise
512–2048Faster wall-clock; needs a scaled learning rate and warm-up
8192+Requires specialised optimisers; used in large-scale pretraining

Note that batch normalisation adds a constraint of its own: with batches below about 8, its statistics turn unreliable and GroupNorm or LayerNorm is the better choice.

Why we process many rows at once

One row at a time is correct and slow. The whole dataset at once is correct and will not fit. All in between is a trade between gradient quality and hardware efficiency, measured here.

example_01.pyNumPy
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
Output
The next block covers the left steps.
example_03.pyNumPy
Output

                

                

                    
Things to try

                    
  1. Watch a single sample flow. Set Batch Size to 1 and press Simulate Flow. One activation pattern moves through the network and one update follows — maximally noisy and maximally frequent.
  2. Batch it up. Set Batch Size to 32 and simulate again. Many samples traverse together and give a single averaged update. The averaging is where the noise reduction comes from.
  3. Count the updates. Fix Epochs and check Batch Size 1 against 100. The same epoch count gives a hundred times fewer weight updates at the more batch — the usual reason a large-batch run appears to underfit.
  4. Widen the network. Raise Input and Output and simulate at a large batch. More arithmetic per sample makes the batched matrix multiply proportionally more worthwhile.

                

                

                    
How batch size affects the result, not just the speed

                    
More batches give a less-variance estimate of the true gradient. So the path to the min is smoother. That sounds strictly good and is not.
The noise in small-batch gradients acts as a regulariser. It shakes the optimiser out of sharp, narrow minima and toward flat, wide ones — and flat minima generalise better, since a small shift in the weights or the data distribution barely changes the loss. Large-batch training tends to settle into sharp minima and often generalises slightly worse, an effect known as the generalisation gap.
The standard mitigation is the linear scaling rule: when you multiply the batch size by k, multiply the learning rate by k as well, since each update is now averaged over k times as much data. Mixed with a short warmup this makes large-batch training match small-batch accuracy in most settings.

                

                

                    
Where this goes wrong

                    
  • Raising the batch size and leaving the learning rate alone. Fewer, no-more updates per epoch means visibly slower convergence. Scale the rate with the batch.
  • Choosing the batch size purely by what fits in memory. The largest batch that fits is not automatically the best one. It is a regularisation choice as well as a throughput one.
  • Batch normalisation with tiny batches. Batch norm estimates mean and variance from the batch. At a batch size of 2 or 4 those estimates are noise. Use group or layer normalisation instead.
  • A final partial batch. When the dataset does not divide evenly the last batch is less. It skews batch-norm statistics and any per-batch averaging. Drop it in training if it is much less.
  • Checking runs by epoch instead of by update. Two runs at other batch sizes have done very other amounts of optimisation after the same count of epochs.

                

                

                    
Key takeaway

                    
Batching turns many small vector-matrix products into one large matrix-matrix product. It is what makes parallel hardware pay off, without changing the net at all. Batch size sets both the count of updates per epoch and the noise in each gradient. So it trades throughput against a regularising effect that favours flat minima. Change it and change the learning rate with it — roughly linearly — or the comparison is not fair.

                

                

                    
Gradient accumulation: a big batch on a small GPU

                    
When the batch you want does not fit in memory, you can simulate it by accumulating gradients over a few small batches before updating:
accum = 4                                   # effective batch = 4 x batch_size
opt.zero_grad()
for i, batch in enumerate(loader):
    loss = criterion(model(batch.x), batch.y) / accum    # scale to keep the mean right
    loss.backward()                                       # gradients accumulate
    if (i + 1) % accum == 0:
        opt.step()
        opt.zero_grad()
Two details matter. The loss must be divided by the accumulation count. Or the effective gradient is accum times too large. And PyTorch accumulates into .grad by default. It is just why this works — the same behaviour that makes forgetting zero_grad() a bug is the mechanism here.
The trade is wall-clock time: four small batches cost about as much as one large one would have. So you get the large-batch statistics without the large-batch speed.

                

                

                    
What limits batch size

                    
Almost always activation memory, not the weights.
Each intermediate activation from the forward pass must be kept for the backward pass. A ResNet-50 at batch size 64 on 224×224 pics stores a few gigabytes of activations, against about 100MB of weights. That is why a net that "fits" can still fail with an out-of-memory error the moment you raise the batch size.
Three ways to buy memory back:
  • Mixed precision stores activations in 16-bit, roughly halving the requirement and speeding up the arithmetic.
  • Gradient checkpointing keeps just some activations and recomputes the rest, trading about 30% more compute for a large memory saving.
  • Gradient accumulation, above, which keeps the memory small and the effective batch large.

                

                

                    
Batching at inference

                    
The same idea applies when serving a net, with a other objective. Batching a few requests together uses the hardware far more well — often five to ten times the throughput of handling them one at a time.
The cost is latency: a request must wait for the batch to fill. Production systems use dynamic batching, collecting requests for a few milliseconds and then handling whatever has arrived. It captures most of the throughput gain with a bounded delay.
One correctness note: batch normalisation must be in eval mode at inference. Or the prediction for one input depends on which other inputs happen to share its batch. That is a genuinely confusing bug — the same input gives other answers — and net.eval() is the fix.

                

                

                    
Questions people ask

                    
What batch size should I use? The largest that fits comfortably, up to about 256 for most work, then scale the learning rate accordingly.
Is a power of two needed? Not needed. And it aligns well with hardware. So 32/64/128/256 are conventional and marginally faster.
Why did my accuracy drop when I raised the batch size? Fewer updates per epoch and an unscaled learning rate. Scale the rate up and add warm-up.
Does batch size affect the final result? Somewhat. Very large batches tend to generalise slightly worse without specialised optimisers. And very small batches are slow and noisy.
Should the last partial batch be dropped? With batch normalisation, yes — a final batch of one or two gives wild statistics. drop_last=True handles it.
Is the loss averaged or summed over a batch? Averaged, by default. Summing makes the effective learning rate depend on batch size.

                

                

                    
Recap in one screen

                    
  • Mini-batches balance gradient quality, hardware efficiency and useful noise. 32–256 is the normal range.
  • An epoch holds dataset / batch_size updates. So a more batch means fewer updates.
  • Scale the learning rate with the batch size, and add warm-up.
  • Activation memory, not the weights, is what limits batch size — mixed precision and checkpointing buy it back.
  • Gradient accumulation gives a large effective batch on small hardware, at the cost of wall-clock time.

                

            


        

    




        
    



Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
  1. What is meant by “Mixed precision” here?
  2. What is meant by “Gradient checkpointing” here?
  3. What is meant by “Gradient accumulation” here?





 Cheat sheet
Batch Processing in Networks
Compare stochastic, mini-batch and full-batch training, and see how batch size trades gradient noise against speed.
DEEP LEARNING · vizlearn.in/deep_learning/batch_processing_in_neural_networks.html
More in Deep Learning
Deep LearningGradient Descent Batch ProcessingDeep LearningFeature Scaling & Weight BiasDeep LearningWeight Initialization MethodsDeep LearningVanishing & Exploding Gradients
Further reading
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact