Quick Context
Deep learning requires massive amounts of mathematical operations (mostly matrix multiplications). To run these efficiently, we use specialized hardware. The choice between a Central Processing Unit (CPU) and a Graphics Processing Unit (GPU) fundamentally changes how data should be batched and processed.
1) The CPU (Latency Optimized)
CPUs are the general-purpose brains of a computer.
- They have a few (e.g., 4 to 24) very powerful cores.
- They are incredibly fast at executing complex, sequential instructions and have almost zero setup time to grab data from main memory (RAM).
- Best for: Processing small batches or single items extremely quickly (e.g., real-time inference at the edge).
2) The GPU (Throughput Optimized)
GPUs were originally designed for rendering graphics but are perfectly suited for deep learning.
- They have thousands of smaller, simpler cores built for SIMD (Single Instruction, Multiple Data) — applying the exact same math operation to massive grids of data simultaneously.
- The Catch: Before the GPU can do math, data must be transferred from system RAM across the PCIe bus into GPU VRAM. This creates a noticeable "overhead" delay.
- Best for: Processing massive batches of data simultaneously (e.g., model training).
3) The Memory Transfer Bottleneck
A GPU is like a massive factory, but it takes time to ship the raw materials (data) there. If you only send one item at a time (batch size 1), the shipping time (transfer overhead) dominates, and the factory sits idle. If you send a massive truckload (batch size 512), the shipping time is easily justified by how fast the factory processes everything at once.
4) Guided Experiments
- Batch Size 1: Set the slider to 1 and run. The CPU wins easily! The GPU's memory transfer overhead takes longer than the CPU takes to just do the math.
- Batch Size 64: Set the slider to 64. The race becomes much closer. The GPU overhead is amortized over more data.
- Batch Size 512: Max out the slider. The GPU crushes the CPU. While the GPU is doing 512 calculations simultaneously, the CPU is stuck looping through small chunks over and over.
- Increase Complexity: Max out Features and Hidden neurons. Watch how the CPU slows down drastically per chunk, giving the GPU an even bigger advantage on large batches.
5) Common Mistakes
- Using small batch sizes on GPUs — If your batch size is too small (e.g., 8 or 16 on a modern GPU), you are vastly underutilizing the GPU cores.
- Data Loading Bottlenecks — Even with a fast GPU, if your CPU cannot load and preprocess images fast enough to feed the GPU, training slows down. This is why PyTorch uses
num_workers.
- Using a GPU for strictly sequential logic — Models like standard RNNs that must process token t before token t+1 struggle to fully saturate a GPU compared to Transformers, which process all tokens in parallel.
6) Key Takeaways
- CPUs excel at fast, sequential processing with low setup latency.
- GPUs excel at massive parallel processing (high throughput) but suffer from memory transfer overhead.
- Always use large batch sizes on GPUs to maximize utilization and amortize transfer times.
- For real-time applications processing one item at a time, a fast CPU or specialized NPU is often better than a heavy desktop GPU.