vizlearn
Home / Deep Learning

Batch Processing in the Neural Network

Deep LearningBy Updated

Interactive architecture builder. Drag to pan, scroll to zoom, right-click to edit.

Overview

A batch is a matrix, not a loop

Feeding one sample through a layer is a vector-matrix product. Feeding 32 samples is a matrix-matrix product — stack the 32 input vectors into a 32×nin matrix and multiply once by the same nin×nout weights.

Crucially the weights are unchanged. Batching does not alter the model; it changes how many samples pass through it per operation. Because the hardware executes one large matrix multiply far more efficiently than 32 small ones, a batch of 32 costs nothing like 32 times a single sample.

Data Rows 100
0%

Analysis

Layers -
Neurons -
Total Params -

Selection

Hover over nodes for details.
Right-click to edit | Scroll to Zoom

Batch Processing in Networks: A Practical Guide

Networks process many samples at once, not one at a time. Batching is what makes GPUs worth using - and the batch size quietly changes both how fast you train and how well the result generalises.

Epoch, batch, iteration

These three get muddled constantly:

  • Batch size — samples processed before one weight update.
  • Iteration (step) — one forward pass, one backward pass, one update.
  • Epoch — one complete pass over the training set.

With 10,000 samples and a batch size of 100, one epoch is 100 iterations, so 10 epochs means 1,000 weight updates. Halving the batch size to 50 doubles the updates per epoch to 200 — which is why batch size and learning rate cannot be tuned independently.

Why examples are processed in groups

Training could update the weights after every single example, or after seeing the entire dataset. Both are worse than the middle option, and for different reasons.

One example at a time gives an extremely noisy gradient estimate and, more importantly, wastes the hardware. A GPU has thousands of cores designed to do the same operation on many values simultaneously; feeding it one 784-element vector uses a fraction of a percent of that capacity.

The entire dataset gives an accurate gradient at enormous cost per update. On a million rows, a single step requires a million forward and backward passes, and the model gets one weight update for all that work.

A mini-batch of 32 to 256 is what everyone uses. The gradient is a reasonable estimate, the matrix multiplications are large enough to saturate the hardware, and the remaining noise is useful — it helps the optimiser escape saddle points.

The arithmetic is a matrix multiply either way: a batch of 64 examples with 784 features is a 64×784 matrix multiplied by the layer's weights, which is one operation rather than 64.

Epochs, steps and the numbers that confuse people

TermMeaning
Batch sizeExamples per update
Step / iterationOne forward pass, one backward pass, one update
EpochOne full pass through the training data
Steps per epochdataset size / batch size

With 50,000 training examples and a batch size of 64, one epoch is 782 steps — 782 weight updates, not one. Ten epochs is 7,820 updates.

That relationship has a consequence people trip over: doubling the batch size halves the number of updates per epoch. So a run with batch size 256 and the same number of epochs performs a quarter of the updates of one with batch size 64, and will usually be behind unless the learning rate is scaled up to compensate.

Batch size, learning rate and generalisation

Larger batches give less noisy gradients, so larger steps are safe. The standard rule is to scale the learning rate linearly with batch size — double the batch, double the rate — with a warm-up period, because the first few large steps are the risky ones.

That rule holds well up to a point. Beyond a few thousand examples per batch, the gradient noise becomes so small that the implicit regularisation it provided disappears, and very large-batch training tends to generalise slightly worse. Techniques such as LARS and LAMB exist specifically to push that ceiling higher.

Batch sizeCharacter
1–8Very noisy; slow on GPUs; sometimes necessary for huge inputs
32–256The normal range; good hardware use, useful noise
512–2048Faster wall-clock; needs a scaled learning rate and warm-up
8192+Requires specialised optimisers; used in large-scale pretraining

Note that batch normalisation adds a constraint of its own: with batches below about 8, its statistics become unreliable and GroupNorm or LayerNorm is the better choice.

Why we process many rows at once

One row at a time is correct and slow. The whole dataset at once is correct and will not fit. Everything in between is a trade between gradient quality and hardware efficiency, measured here.

example_01.pyNumPy
Output

                

                

                    
Things to try

                    
  1. Watch a single sample flow. Set Batch Size to 1 and press Simulate Flow. One activation pattern moves through the network and one update follows — maximally noisy and maximally frequent.
  2. Batch it up. Set Batch Size to 32 and simulate again. Many samples traverse together and produce a single averaged update. The averaging is where the noise reduction comes from.
  3. Count the updates. Fix Epochs and compare Batch Size 1 against 100. The same epoch count gives a hundred times fewer weight updates at the larger batch — the usual reason a large-batch run appears to underfit.
  4. Widen the network. Raise Input and Output and simulate at a large batch. More arithmetic per sample makes the batched matrix multiply proportionally more worthwhile.

                

                

                    
How batch size affects the result, not just the speed

                    
Larger batches give a lower-variance estimate of the true gradient, so the path to the minimum is smoother. That sounds strictly good and is not.
The noise in small-batch gradients acts as a regulariser. It shakes the optimiser out of sharp, narrow minima and toward flat, wide ones — and flat minima generalise better, because a small shift in the weights or the data distribution barely changes the loss. Large-batch training tends to settle into sharp minima and frequently generalises slightly worse, an effect known as the generalisation gap.
The standard mitigation is the linear scaling rule: when you multiply the batch size by k, multiply the learning rate by k as well, since each update is now averaged over k times as much data. Combined with a short warmup this makes large-batch training match small-batch accuracy in most settings.

                

                

                    
Where this goes wrong

                    
  • Raising the batch size and leaving the learning rate alone. Fewer, no-larger updates per epoch means visibly slower convergence. Scale the rate with the batch.
  • Choosing the batch size purely by what fits in memory. The largest batch that fits is not automatically the best one; it is a regularisation choice as well as a throughput one.
  • Batch normalisation with tiny batches. Batch norm estimates mean and variance from the batch. At a batch size of 2 or 4 those estimates are noise; use group or layer normalisation instead.
  • A final partial batch. When the dataset does not divide evenly the last batch is smaller, which skews batch-norm statistics and any per-batch averaging. Drop it during training if it is much smaller.
  • Comparing runs by epoch instead of by update. Two runs at different batch sizes have done very different amounts of optimisation after the same number of epochs.

                

                

                    
Key takeaway

                    
Batching turns many small vector-matrix products into one large matrix-matrix product, which is what makes parallel hardware pay off, without changing the model at all. Batch size sets both the number of updates per epoch and the noise in each gradient, so it trades throughput against a regularising effect that favours flat minima. Change it and change the learning rate with it — roughly linearly — or the comparison is not fair.

                

                

                    
Gradient accumulation: a big batch on a small GPU

                    
When the batch you want does not fit in memory, you can simulate it by accumulating gradients over several small batches before updating:
accum = 4                                   # effective batch = 4 x batch_size
opt.zero_grad()
for i, batch in enumerate(loader):
    loss = criterion(model(batch.x), batch.y) / accum    # scale to keep the mean right
    loss.backward()                                       # gradients accumulate
    if (i + 1) % accum == 0:
        opt.step()
        opt.zero_grad()
Two details matter. The loss must be divided by the accumulation count, or the effective gradient is accum times too large. And PyTorch accumulates into .grad by default, which is exactly why this works — the same behaviour that makes forgetting zero_grad() a bug is the mechanism here.
The trade is wall-clock time: four small batches cost about as much as one large one would have, so you get the large-batch statistics without the large-batch speed.

                

                

                    
What limits batch size

                    
Almost always activation memory, not the weights.
Every intermediate activation from the forward pass must be kept for the backward pass. A ResNet-50 at batch size 64 on 224×224 images stores several gigabytes of activations, against about 100MB of weights. That is why a model that "fits" can still fail with an out-of-memory error the moment you raise the batch size.
Three ways to buy memory back:
  • Mixed precision stores activations in 16-bit, roughly halving the requirement and speeding up the arithmetic.
  • Gradient checkpointing keeps only some activations and recomputes the rest, trading about 30% more compute for a large memory saving.
  • Gradient accumulation, above, which keeps the memory small and the effective batch large.

                

                

                    
Batching at inference

                    
The same idea applies when serving a model, with a different objective. Batching several requests together uses the hardware far more efficiently — often five to ten times the throughput of processing them one at a time.
The cost is latency: a request must wait for the batch to fill. Production systems use dynamic batching, collecting requests for a few milliseconds and then processing whatever has arrived, which captures most of the throughput gain with a bounded delay.
One correctness note: batch normalisation must be in eval mode at inference, or the prediction for one input depends on which other inputs happen to share its batch. That is a genuinely confusing bug — the same input gives different answers — and model.eval() is the fix.

                

                

                    
Questions people ask

                    
What batch size should I use? The largest that fits comfortably, up to about 256 for most work, then scale the learning rate accordingly.
Is a power of two required? Not required, and it aligns well with hardware, so 32/64/128/256 are conventional and marginally faster.
Why did my accuracy drop when I increased the batch size? Fewer updates per epoch and an unscaled learning rate. Scale the rate up and add warm-up.
Does batch size affect the final result? Somewhat. Very large batches tend to generalise slightly worse without specialised optimisers, and very small batches are slow and noisy.
Should the last partial batch be dropped? With batch normalisation, yes — a final batch of one or two gives wild statistics. drop_last=True handles it.
Is the loss averaged or summed over a batch? Averaged, by default. Summing makes the effective learning rate depend on batch size.

                

                

                    
Recap in one screen

                    
  • Mini-batches balance gradient quality, hardware efficiency and useful noise; 32–256 is the normal range.
  • An epoch contains dataset / batch_size updates, so a larger batch means fewer updates.
  • Scale the learning rate with the batch size, and add warm-up.
  • Activation memory, not the weights, is what limits batch size — mixed precision and checkpointing buy it back.
  • Gradient accumulation gives a large effective batch on small hardware, at the cost of wall-clock time.

                

            


        

    




        
    



Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
  1. What is meant by “Mixed precision” here?
  2. What is meant by “Gradient checkpointing” here?
  3. What is meant by “Gradient accumulation” here?





 Cheat sheet
Batch Processing in Networks
Feeding one sample through a layer is a vector-matrix product. Feeding 32 samples is a matrix-matrix product — stack the 32 input vectors into a 32×nin matrix and multiply once by the same nin×nout weights.
DEEP LEARNING · vizlearn.in/deep_learning/batch_processing_in_neural_networks.html
More in Deep Learning
Deep LearningGradient Descent Batch ProcessingDeep LearningFeature Scaling & Weight BiasDeep LearningWeight Initialization MethodsDeep LearningVanishing & Exploding Gradients
Further reading
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact