Watch a head-to-head race! The CPU has low latency but processes sequentially. The GPU has a memory-transfer delay, but processes massive batches in parallel. Adjust the batch size to see who wins.
Overview
Latency versus throughput
A CPU has a handful of cores optimised to finish one instruction stream as fast as possible: high clock speeds, deep pipelines, large caches, aggressive branch prediction. It is built for latency.
A GPU has thousands of much simpler cores running in lockstep on different data. Any single core is slower than a CPU core, and there are so many that the aggregate arithmetic throughput is an order of magnitude higher. It is built for throughput.
Neural network training is almost entirely matrix multiplication, which is the ideal case for that design: every output element is an independent dot product, so there is nothing to serialise and nothing to predict.
64
CPU
Sequential
GPU
Parallel
CPU Time0.0 ms
Idle
VS
Winner
CPU
GPU Time0.0 ms
Idle
Model Training on CPU vs GPU: A Practical Guide
A GPU is not a faster CPU. It is a wider one - thousands of slow cores instead of a few fast ones - and that shape decides which workloads it transforms and which it does not.
Why batch size is the variable that matters
Multiplying a 1×512 input by a 512×512 weight matrix uses a tiny fraction of a GPU’s cores; the rest sit idle. Multiply a 256×512 batch by the same weights and the work grows 256-fold while the time barely moves, because that work fills capacity that was already there and already paid for.
This is why batch size dominates GPU utilisation. Below a threshold the GPU is latency-bound and a CPU can genuinely be competitive; above it, throughput takes over and the GPU wins by a wide margin. The same reasoning explains why wider layers use a GPU better than narrow ones.
Why a GPU is faster for this particular job
A CPU has a handful of powerful cores optimised for doing different things quickly, one after another. A GPU has thousands of simple cores optimised for doing the same thing to many values simultaneously.
Neural network training is almost entirely matrix multiplication, which is exactly that pattern: multiply and add, thousands of independent times, with no branching. It maps onto GPU hardware nearly perfectly.
The gap in practice:
Task
CPU
GPU
Small tabular network
Seconds
Seconds — no benefit
ResNet-50 on ImageNet, one epoch
Days
Minutes
Fine-tuning BERT
Days
~1 hour
Inference, single request
Often fine
Faster, and often unnecessary
The pattern is that GPUs win when the model is large and the batches are large enough to keep thousands of cores busy. For a two-layer network on 10,000 rows, the overhead of moving data to the device can make the GPU slower.
What actually limits GPU training
Rarely the arithmetic. Three other things usually bind first.
Memory. Activations from the forward pass must be kept for the backward pass, and that dominates — a ResNet-50's weights are about 100MB while its activations at batch size 64 run to several gigabytes. This is why out-of-memory errors arrive when you raise the batch size, not when you load the model.
The data loader. If the CPU cannot decode and augment images fast enough, the GPU idles. Watch nvidia-smi: utilisation oscillating between 100% and near zero means the loader is the bottleneck, and more num_workers is the fix.
Data transfer. Moving tensors between host and device costs time. Keep them on the GPU, use pin_memory=True, and avoid .cpu() or .item() calls inside the training loop — each one forces a synchronisation that stalls the pipeline.
Mixed precision: the free speed-up
Modern GPUs have dedicated hardware for 16-bit matrix multiplication, several times faster than 32-bit. Mixed precision uses 16-bit for the heavy arithmetic and keeps 32-bit copies of the weights for accuracy.
scaler = torch.cuda.amp.GradScaler()
for batch in loader:
opt.zero_grad()
with torch.autocast("cuda", dtype=torch.float16):
loss = criterion(model(batch.x), batch.y)
scaler.scale(loss).backward()
scaler.step(opt)
scaler.update()
Two to three times faster and roughly half the memory, with essentially no accuracy cost. It should be the default for any GPU training.
The GradScaler exists for a specific reason: 16-bit numbers underflow to zero below about 6e-8, and many gradients are smaller than that. Scaling the loss up before the backward pass and unscaling afterwards keeps them representable. On newer hardware, bfloat16 has a wider exponent range and needs no scaler at all.
Interactive Exploration Guide
Start where the CPU can win. Set Batch Size to 1 and press Start Race. With one sample the GPU’s parallelism is unused and the launch overhead dominates — the CPU is competitive or ahead.
Find the crossover. Raise Batch Size to 32 and race again, then 128. Somewhere in that range the GPU pulls decisively ahead; below it the two are close.
Push it wide. Set Batch Size to 512. GPU time barely increases while CPU time scales roughly linearly — that flatness is unused capacity being filled rather than extra speed appearing.
Widen the layer instead. Raise Hidden to 32 and keep the batch small. More arithmetic per sample helps the GPU too, for the same reason: the bottleneck is occupancy, not the amount of work.
The transfer cost nobody budgets for
CPU and GPU have separate memory. Every tensor must cross the PCIe bus, and that bus is slow relative to on-device bandwidth — roughly 16–32 GB/s against 900+ GB/s of GPU memory bandwidth.
So a computation is only worth moving to the GPU if the arithmetic saved exceeds the transfer cost. Copying a batch across to run one cheap elementwise operation is slower than doing it on the CPU. This is why the standard advice is to move the model and the batch to the device once and keep everything there for the whole step, rather than moving tensors back and forth — and why an innocuous .cpu() or .item() inside a training loop can dominate the profile, since it also forces a synchronisation that stalls the pipeline.
What trips people up
Batch too small to occupy the device. The most common reason a GPU shows 15% utilisation. If memory allows, increase the batch before blaming the hardware.
A data loader that cannot keep up. If preprocessing happens on one CPU thread, the GPU waits. Use multiple worker processes and pinned memory.
Synchronising inside the loop. Calling .item() on the loss every step forces the CPU to wait for the GPU. Accumulate on device and read occasionally.
Expecting a speedup on small models. For a small network on tabular data the CPU is often faster outright — there is not enough parallel work to amortise the launch and transfer overhead.
In one line
A GPU trades per-core speed for thousands of cores, which suits neural networks because matrix multiplication is embarrassingly parallel. The speedup is real only when there is enough work in flight to occupy the device, so batch size and layer width matter more than anything else, and the separate memory space means transfers and synchronisations can quietly become the bottleneck. Small model, small batch: the CPU may well win.
When the CPU is the right choice
Not everything needs a GPU, and reaching for one reflexively wastes money.
Tabular data with gradient boosting. XGBoost and LightGBM on CPU are extremely fast and frequently beat neural networks on this data anyway.
Small networks. Below a few hundred thousand parameters and with modest batches, the transfer overhead can exceed the compute saving.
Inference at low volume. A single prediction from a modest model takes milliseconds on a CPU. Many production systems serve models on CPU deliberately: no GPU to provision, simpler deployment, lower cost.
Data preprocessing. Loading, decoding and augmenting are CPU work, and that is where the parallelism should go.
The practical rule: profile before buying. If GPU utilisation sits at 20%, the answer is a better data pipeline, not a bigger GPU.
Scaling beyond one device
Data parallelism is the common approach: each GPU holds a full copy of the model, processes a different slice of the batch, and the gradients are averaged across devices. Use DistributedDataParallel rather than the older DataParallel, which is significantly slower.
Note that the effective batch size is multiplied by the number of GPUs, so the learning rate needs scaling accordingly.
Model parallelism splits the model itself across devices, and is only needed when the model does not fit on one. Pipeline parallelism splits it by layers and streams micro-batches through. Sharded training (ZeRO, FSDP) splits the optimiser state, gradients and parameters across devices, which is what makes training models with tens of billions of parameters possible — Adam's state alone is twice the model size.
For most work, one GPU with mixed precision and a well-tuned data loader is enough, and reaching for distributed training before saturating a single device is premature.
What a GPU is actually better at
A GPU is not a fast CPU. It is thousands of slow cores, which makes it enormously better at some shapes of work and no better at all at others -- and that distinction decides how you write training code.
example_01.pyNumPy
import numpy as np
import time
rng = np.random.default_rng(0)
print("the arithmetic in a training step, counted. one dense layer,")
print("batch B, input D, output H, needs 2*B*D*H floating point operations")
print("in the forward pass and roughly twice that backward:")
print()
print("%8s %8s %8s %16s %16s" % ("B", "D", "H", "forward FLOPs", "fwd+bwd"))
for B, D, H in ((1, 512, 512), (32, 512, 512), (512, 512, 512),
(512, 4096, 4096)):
f = 2 * B * D * H
print("%8d %8d %8d %16s %16s" % (B, D, H, "%.2e" % f, "%.2e" % (3 * f)))
print()
print("a modern GPU does something like 1e14 of those per second and a CPU")
print("core perhaps 1e10. but that peak is only reachable if there is enough")
print("independent work to fill every core at once.")
print()
print("here is that dependency, measured in this browser. the SAME NUMBER OF")
print("FLOPs every row -- a side x side matmul costs 2 * side^3, so the call")
print("count is chosen to hold the total arithmetic fixed:")
print()
print("%14s %10s %14s %14s %12s"
% ("shape", "calls", "total FLOPs", "seconds", "relative"))
BUDGET = 2e8
first = None
for side in (8, 32, 128, 256):
per_call = 2 * side ** 3
n_calls = max(int(BUDGET // per_call), 1)
A = rng.normal(size=(side, side)).astype(np.float32)
B = rng.normal(size=(side, side)).astype(np.float32)
t0 = time.time()
for _ in range(n_calls):
_ = A @ B
t = time.time() - t0
first = first or t
print("%14s %10d %14s %14.4f %11.2fx"
% ("%dx%d" % (side, side), n_calls, "%.2e" % (n_calls * per_call),
t, t / first))
print()
print("identical arithmetic, and the small-block version takes several times")
print("longer. the cost is per-CALL overhead and poor cache reuse, neither of")
print("which the FLOP count can see.")
print()
print("even on two CPU cores that gap is real. on a GPU it is far wider,")
print("because an 8x8 matmul leaves essentially the whole chip idle while a")
print("512x512 one can fill it.")
print()
print("what this means in practice, as a list of consequences:")
print()
print(" BATCH SIZE. a batch of 1 on a GPU is close to a waste of the GPU.")
print(" this is the single biggest reason batching exists.")
print()
print(" DATA TRANSFER. moving arrays between host and device costs real")
print(" time, and a step that transfers per-batch can spend more time")
print(" moving than computing:")
for mb in (1, 16, 256):
print(" %4d MB at ~10 GB/s over PCIe: %6.2f ms each way"
% (mb, mb / 10_000 * 1000))
print(" which is why you pin memory, prefetch, and keep the model resident.")
print()
print(" PRECISION. a GPU's tensor cores run float16 and bfloat16 several")
print(" times faster than float32. the same 128x128 matmul, three ways:")
for dt in (np.float64, np.float32, np.float16):
A = rng.normal(size=(128, 128)).astype(dt)
t0 = time.time()
for _ in range(5):
_ = A @ A
print(" %-10s %8.4f s (%d bytes per number)"
% (np.dtype(dt).name, time.time() - t0, np.dtype(dt).itemsize))
print(" do not read the float16 row as an argument against it -- numpy")
print(" has no float16 hardware path and emulates it in software, so it")
print(" comes out SLOWEST here. on a GPU with tensor cores that row is")
print(" several times faster than float32, and that gap is exactly what")
print(" mixed precision training is buying.")
print()
print(" WHAT A GPU DOES NOT HELP WITH. python loops, data loading and")
print(" preprocessing, small sequential operations, and anything that has")
print(" to synchronise every step. a training loop that is slow because of")
print(" its dataloader will be exactly as slow on a better GPU.")
print()
print("the diagnosis is always the same: measure whether you are compute")
print("bound, memory bound, or input bound BEFORE buying anything. a GPU at")
print("30 percent utilisation is telling you the bottleneck is somewhere else.")
Output
Questions people ask
How much GPU memory do I need? 8GB trains most vision models at modest batch sizes; 24GB is comfortable; large language model fine-tuning wants 40GB or more, or parameter-efficient methods like LoRA.
Why is my GPU only 30% utilised? The data loader, almost always. Raise num_workers, pre-resize images, and check for .item() calls in the loop.
Does mixed precision hurt accuracy? In practice no, with the gradient scaler in place. It is standard.
Can I train on Apple Silicon? Yes — PyTorch's MPS backend uses the integrated GPU. Slower than a discrete card, and much faster than CPU for medium models.
Should I use TPUs? They are excellent for very large-scale training on Google's stack, and involve more code changes than moving between GPUs.
Is more VRAM or a faster GPU better? VRAM decides what you can train; speed decides how long it takes. Memory is usually the binding constraint first.
Recap in one screen
GPUs win because training is mostly matrix multiplication, which parallelises almost perfectly.
Activation memory, not the weights, is what limits batch size and causes out-of-memory errors.
An idle GPU usually means a starved data loader — check utilisation before blaming the hardware.
Mixed precision gives 2–3× speed and half the memory for essentially no accuracy cost; use it by default.
Small models, tabular data and low-volume inference are all legitimately CPU work.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
A GPU trades per-core speed for thousands of cores, which suits neural networks because matrix multiplication is embarrassingly parallel. The speedup is real only when there is enough work in flight to occupy the device, so batch size and layer width matter more than anything else, and the separate memory space means transfers and synchronisations can quietly become the bottleneck.
What does this module say about “Latency versus throughput”?
A CPU has a handful of cores optimised to finish one instruction stream as fast as possible: high clock speeds, deep pipelines, large caches, aggressive branch prediction. It is built for latency.
What does this module say about “Why batch size is the variable that matters”?
Multiplying a 1×512 input by a 512×512 weight matrix uses a tiny fraction of a GPU’s cores; the rest sit idle. Multiply a 256×512 batch by the same weights and the work grows 256-fold while the time barely moves, because that work fills capacity that was already there and already paid for.
Cheat sheet
Model Training on CPU vs GPU
Watch a head-to-head race! The CPU has low latency but processes sequentially. The GPU has a memory-transfer delay, but processes massive batches in parallel. Adjust the batch size to see who wins.
DEEP LEARNING · vizlearn.in/deep_learning/model_training_on_cpu_vs_gpu.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.