map / ai-hardware / gpu-vs-cpu
AI Hardware · principle 2 of 7
GPU vs CPU
CPUs spend silicon on making one thread fast; GPUs spend it on running thousands of simple threads at once.
01 · Why it’s true
A CPU has a small number of powerful cores built to run varied, branching code with low latency, backed by large caches and features that keep a single thread fast.
A GPU has thousands of simpler arithmetic units that run the same instruction across many pieces of data at once. Neural networks are mostly large matrix multiplications — the same operation over huge arrays — so they map naturally onto GPUs. Code with lots of branches, or work that’s mostly sequential, doesn’t.
02 · Picture it
03 · In the interview
- Explaining why ML workloads run on GPUs or other accelerators.
- Spotting CPU-side bottlenecks — data loading, preprocessing — that leave GPUs idle.
- Understanding that a GPU server is still a computer with CPUs, memory, disks, and network that can fail.
04 · Practice question
“GPU utilization on our training job swings between 100% and near zero every few seconds. What would you look at?”
A strong answer covers
- The GPUs are probably waiting on input: data loading or preprocessing on the CPU can’t keep up.
- Check CPU usage of the data loader workers, storage or network read throughput, and whether batches are prefetched.
- Periodic stalls can also be checkpoint writes or synchronization between GPUs.
- Fixes: more loader workers, prefetching, caching data locally, faster formats, overlapping I/O with compute.