Watch a head-to-head race! The CPU has low latency but processes sequentially. The GPU has a memory-transfer delay, but processes massive batches in parallel. Adjust the batch size to see who wins.
64
CPU
Sequential
GPU
Parallel
CPU Time0.0 ms
Idle
VS
Winner
CPU
GPU Time0.0 ms
Idle
Overview
Latency versus throughput
A CPU has a handful of cores optimised to finish one instruction stream as fast as possible: high clock speeds, deep pipelines, large caches, aggressive branch prediction. It is built for latency.
A GPU has thousands of much simpler cores running in lockstep on other data. Any single core is slower than a CPU core. And there are so many that the aggregate arithmetic throughput is an order of magnitude more. It is built for throughput.
Neural network training is almost fully matrix multiplication. It is the ideal case for that design: each output item is an independent dot product. So there is nothing to serialise and nothing to predict. The speedup is not free — data must be transferred to GPU memory over a PCIe or NVLink bus. And that transfer is the bottleneck when the computation per batch is small. Batch size is therefore the var that decides whether a GPU helps: below a threshold the cores sit idle waiting for data and a CPU can genuinely compete. Above it, throughput dominates and the GPU wins by an order of magnitude. This page lets you sweep batch size and net width on both devices, shows where the crossover happens. And covers mixed-precision training and data-loading plans that keep the GPU fed.
A GPU is not a faster CPU. It is a wider one - thousands of slow cores instead of a few fast ones - and that shape decides which workloads it transforms and which it does not.
Latency versus throughput
A CPU has a handful of cores optimised to finish one instruction stream as fast as possible: high clock speeds, deep pipelines, large caches, aggressive branch prediction. It is built for latency.
A GPU has thousands of much simpler cores running in lockstep on other data. Any single core is slower than a CPU core. And there are so many that the aggregate arithmetic throughput is an order of magnitude more. It is built for throughput.
Neural network training is almost fully matrix multiplication. It is the ideal case for that design: each output item is an independent dot product. So there is nothing to serialise and nothing to predict. The speedup is not free — data must be transferred to GPU memory over a PCIe or NVLink bus. And that transfer is the bottleneck when the computation per batch is small. Batch size is therefore the var that decides whether a GPU helps: below a threshold the cores sit idle waiting for data and a CPU can genuinely compete. Above it, throughput dominates and the GPU wins by an order of magnitude. This page lets you sweep batch size and net width on both devices, shows where the crossover happens. And covers mixed-precision training and data-loading plans that keep the GPU fed.
Why batch size is the var that matters
Multiplying a 1×512 input by a 512×512 weight matrix uses a tiny fraction of a GPU’s cores. The rest sit idle. Multiply a 256×512 batch by the same weights and the work grows 256-fold while the time barely moves, since that work fills capacity that was now there and now paid for.
This is why batch size dominates GPU utilisation. Below a threshold the GPU is latency-bound and a CPU can genuinely be strong. Above it, throughput takes over and the GPU wins by a wide margin. The same reasoning explains why wider layers use a GPU better than narrow ones.
Why a GPU is faster for this particular job
A CPU has a handful of powerful cores optimised for doing other things quickly, one after another. A GPU has thousands of simple cores optimised for doing the same thing to many vals at once.
Neural network training is almost fully matrix multiplication. It is just that pattern: multiply and add, thousands of independent times, with no branching. It maps onto GPU hardware nearly perfectly.
The gap in practice:
Task
CPU
GPU
Small tabular network
Seconds
Seconds — no benefit
ResNet-50 on ImageNet, one epoch
Days
Minutes
Fine-tuning BERT
Days
~1 hour
Inference, single request
Often fine
Faster, and often unnecessary
The pattern is that GPUs win when the net is large and the batches are large enough to keep thousands of cores busy. For a two-layer network on 10,000 rows, the cost of moving data to the device can make the GPU slower.
What really limits GPU training
Rarely the arithmetic. Three other things often bind first.
Memory. Activations from the forward pass must be kept for the backward pass. And that dominates — a ResNet-50's weights are about 100MB while its activations at batch size 64 run to a few gigabytes. This is why out-of-memory errors arrive when you raise the batch size, not when you load the net.
The data loader. If the CPU cannot decode and augment pics fast enough, the GPU idles. Watch nvidia-smi: utilisation oscillating between 100% and near zero means the loader is the bottleneck. And more num_workers is the fix.
Data transfer. Moving tensors between host and device costs time. Keep them on the GPU, use pin_memory=True. And avoid .cpu() or .item() calls inside the training loop — each one forces a synchronisation that stalls the pipeline.
Mixed precision: the free speed-up
Modern GPUs have dedicated hardware for 16-bit matrix multiplication, a few times faster than 32-bit. Mixed precision uses 16-bit for the heavy arithmetic and keeps 32-bit copies of the weights for accuracy.
scaler = torch.cuda.amp.GradScaler()
for batch in loader:
opt.zero_grad()
with torch.autocast("cuda", dtype=torch.float16):
loss = criterion(model(batch.x), batch.y)
scaler.scale(loss).backward()
scaler.step(opt)
scaler.update()
Two to three times faster and roughly half the memory, with in effect no accuracy cost. It should be the default for any GPU training.
The GradScaler exists for a set reason: 16-bit counts underflow to zero below about 6e-8. And many gradients are less than that. Scaling the loss up before the backward pass and unscaling afterwards keeps them representable. On newer hardware, bfloat16 has a wider exponent range and needs no scaler at all.
Interactive Exploration Guide
Start where the CPU can win. Set Batch Size to 1 and press Start Race. With one sample the GPU’s parallelism is unused and the launch cost dominates — the CPU is strong or ahead.
Find the crossover. Raise Batch Size to 32 and race again, then 128. Somewhere in that range the GPU pulls decisively ahead. Below it the two are close.
Push it wide. Set Batch Size to 512. GPU time barely raises while CPU time scales roughly linearly — that flatness is unused capacity being filled rather than extra speed appearing.
Widen the layer instead. Raise Hidden to 32 and keep the batch small. More arithmetic per sample helps the GPU too, for the same reason: the bottleneck is occupancy, not the amount of work.
The transfer cost nobody budgets for
CPU and GPU have split memory. Each tensor must cross the PCIe bus. And that bus is slow relative to on-device bandwidth — roughly 16–32 GB/s against 900+ GB/s of GPU memory bandwidth.
So a computation is just worth moving to the GPU if the arithmetic saved exceeds the transfer cost. Copying a batch across to run one cheap elementwise step is slower than doing it on the CPU. This is why the standard advice is to move the net and the batch to the device once and keep all there for the whole step, rather than moving tensors back and forth — and why an innocuous .cpu() or .item() inside a training loop can dominate the profile, since it also forces a synchronisation that stalls the pipeline.
What trips people up
Batch too small to occupy the device. The most common reason a GPU shows 15% utilisation. If memory lets, increase the batch before blaming the hardware.
A data loader that cannot keep up. If preprocessing happens on one CPU thread, the GPU waits. Use multiple worker processes and pinned memory.
Synchronising inside the loop. Calling .item() on the loss each step forces the CPU to wait for the GPU. Accumulate on device and read at times.
Expecting a speedup on small nets. For a small network on tabular data the CPU is often faster outright — there is not enough parallel work to amortise the launch and transfer cost.
In one line
A GPU trades per-core speed for thousands of cores. It suits neural networks since matrix multiplication is embarrassingly parallel. The speedup is real just when there is enough work in flight to occupy the device. So batch size and layer width matter more than anything else. And the split memory space means transfers and synchronisations can quietly turn the bottleneck. Small net, small batch: the CPU may well win.
When the CPU is the right choice
Not all needs a GPU, and reaching for one reflexively wastes money.
Tabular data with gradient boosting. XGBoost and LightGBM on CPU are very fast and often beat neural networks on this data anyway.
Small networks. Below a few hundred thousand params and with modest batches, the transfer cost can exceed the compute saving.
Inference at low volume. A single prediction from a modest net takes milliseconds on a CPU. Many production systems serve nets on CPU deliberately: no GPU to provision, simpler deployment, less cost.
Data preprocessing. Loading, decoding and augmenting are CPU work. And that is where the parallelism should go.
The real rule: profile before buying. If GPU utilisation sits at 20%, the answer is a better data pipeline, not a bigger GPU.
Scaling beyond one device
Data parallelism is the common way: each GPU holds a full copy of the net, processes a other slice of the batch. And the gradients are averaged across devices. Use DistributedDataParallel rather than the older DataParallel, which is much slower.
Note that the effective batch size is multiplied by the count of GPUs. So the learning rate needs scaling accordingly.
Net parallelism splits the net it across devices. And is just needed when the net does not fit on one. Pipeline parallelism splits it by layers and streams micro-batches through. Sharded training (ZeRO, FSDP) splits the optimiser state, gradients and params across devices. It is what makes training nets with tens of billions of params possible — Adam's state alone is twice the net size.
For most work, one GPU with mixed precision and a well-tuned data loader is enough. And reaching for distributed training before saturating a single device is premature.
What a GPU is really better at
A GPU is not a fast CPU. It is thousands of slow cores. It makes it enormously better at some shapes of work and no better at all at others -- and that distinction decides how you write training code.
example_01.pyNumPy
import numpy as np
import time
rng = np.random.default_rng(0)
print("the arithmetic in a training step, counted. one dense layer,")
print("batch B, input D, output H, needs 2*B*D*H floating point operations")
print("in the forward pass and roughly twice that backward:")
print()
print("%8s %8s %8s %16s %16s" % ("B", "D", "H", "forward FLOPs", "fwd+bwd"))
for B, D, H in ((1, 512, 512), (32, 512, 512), (512, 512, 512),
(512, 4096, 4096)):
f = 2 * B * D * H
print("%8d %8d %8d %16s %16s" % (B, D, H, "%.2e" % f, "%.2e" % (3 * f)))
print()
print("a modern GPU does something like 1e14 of those per second and a CPU")
print("core perhaps 1e10. but that peak is only reachable if there is enough")
print("independent work to fill every core at once.")
print()
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
import time
print("here is that dependency, measured in this browser. the SAME NUMBER OF")
print("FLOPs every row -- a side x side matmul costs 2 * side^3, so the call")
print("count is chosen to hold the total arithmetic fixed:")
print()
print("%14s %10s %14s %14s %12s"
% ("shape", "calls", "total FLOPs", "seconds", "relative"))
BUDGET = 2e8
first = None
for side in (8, 32, 128, 256):
per_call = 2 * side ** 3
n_calls = max(int(BUDGET // per_call), 1)
A = rng.normal(size=(side, side)).astype(np.float32)
B = rng.normal(size=(side, side)).astype(np.float32)
t0 = time.time()
for _ in range(n_calls):
_ = A @ B
t = time.time() - t0
first = first or t
print("%14s %10d %14s %14.4f %11.2fx"
% ("%dx%d" % (side, side), n_calls, "%.2e" % (n_calls * per_call),
t, t / first))
print()
print("identical arithmetic, and the small-block version takes several times")
print("longer. the cost is per-CALL overhead and poor cache reuse, neither of")
print("which the FLOP count can see.")
print()
print("even on two CPU cores that gap is real. on a GPU it is far wider,")
print("because an 8x8 matmul leaves in effect the whole chip idle while a")
print("512x512 one can fill it.")
print()
print("what this means in practice, as a list of consequences:")
Output
The next block covers the left steps.
example_03.pyNumPy
print()
print(" BATCH SIZE. a batch of 1 on a GPU is close to a waste of the GPU.")
print(" this is the single biggest reason batching exists.")
print()
print(" DATA TRANSFER. moving arrays between host and device costs real")
print(" time, and a step that transfers per-batch can spend more time")
print(" moving than computing:")
for mb in (1, 16, 256):
print(" %4d MB at ~10 GB/s over PCIe: %6.2f ms each way"
% (mb, mb / 10_000 * 1000))
print(" which is why you pin memory, prefetch, and keep the model resident.")
print()
print(" PRECISION. a GPU's tensor cores run float16 and bfloat16 several")
print(" times faster than float32. the same 128x128 matmul, three ways:")
for dt in (np.float64, np.float32, np.float16):
A = rng.normal(size=(128, 128)).astype(dt)
t0 = time.time()
for _ in range(5):
_ = A @ A
print(" %-10s %8.4f s (%d bytes per number)"
% (np.dtype(dt).name, time.time() - t0, np.dtype(dt).itemsize))
print(" do not read the float16 row as an argument against it -- numpy")
print(" has no float16 hardware path and emulates it in software, so it")
print(" comes out SLOWEST here. on a GPU with tensor cores that row is")
print(" several times faster than float32, and that gap is exactly what")
print(" mixed precision training is buying.")
print()
print(" WHAT A GPU DOES NOT HELP WITH. python loops, data loading and")
print(" preprocessing, small sequential operations, and anything that has")
print(" to synchronise every step. a training loop that is slow because of")
print(" its dataloader will be exactly as slow on a better GPU.")
print()
print("the diagnosis is always the same: measure whether you are compute")
print("bound, memory bound, or input bound BEFORE buying anything. a GPU at")
print("30 percent utilisation is telling you the bottleneck is somewhere else.")
Output
Questions people ask
How much GPU memory do I need? 8GB trains most vision nets at modest batch sizes. 24GB is comfortable. Large language net fine-tuning wants 40GB or more. Or param-fast ways like LoRA.
Why is my GPU just 30% utilised? The data loader, almost always. Raise num_workers, pre-resize pics, and check for .item() calls in the loop.
Does mixed precision hurt accuracy? In practice no, with the gradient scaler in place. It is standard.
Can I train on Apple Silicon? Yes — PyTorch's MPS backend uses the integrated GPU. Slower than a discrete card. And much faster than CPU for medium nets.
Should I use TPUs? They are excellent for very large-scale training on Google's stack. And involve more code changes than moving between GPUs.
Is more VRAM or a faster GPU better? VRAM decides what you can train. Speed decides how long it takes. Memory is often the binding constraint first.
Recap in one screen
GPUs win since training is mostly matrix multiplication, which parallelises almost perfectly.
Activation memory, not the weights, is what limits batch size and causes out-of-memory errors.
An idle GPU often means a starved data loader — check utilisation before blaming the hardware.
Mixed precision gives 2–3× speed and half the memory for in effect no accuracy cost. Use it by default.
Small nets, tabular data and low-volume inference are all legitimately CPU work.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “Latency versus throughput”?
What does this module say about “Why batch size is the var that matters”?
Cheat sheet
Model Training on CPU vs GPU
Sweep batch size from 1 to 256 and watch the GPU crossover: too small and the CPU wins, large enough and thousands of cores pay off.
DEEP LEARNING · vizlearn.in/deep_learning/model_training_on_cpu_vs_gpu.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.