map / ai-hardware / gpu-vs-cpu

AI Hardware · principle 2 of 7

GPU vs CPU

CPUs spend silicon on making one thread fast; GPUs spend it on running thousands of simple threads at once.

01 · Why it’s true

A CPU has a small number of powerful cores built to run varied, branching code with low latency, backed by large caches and features that keep a single thread fast.

A GPU has thousands of simpler arithmetic units that run the same instruction across many pieces of data at once. Neural networks are mostly large matrix multiplications — the same operation over huge arrays — so they map naturally onto GPUs. Code with lots of branches, or work that’s mostly sequential, doesn’t.

02 · Picture it

CPU: a few strong coresOptimized for one thread’s latencyGood at branching, varied codeLarge caches per coreRuns the OS, control logic, dataloadingGPU: thousands of simple unitsOptimized for total throughputSame operation over lots of dataVery high memory bandwidthRuns the matrix math of training andinferencevsNeural networks are mostly big matrix multiplications, which is exactly the work GPUs arebuilt for.
CPU: a few strong coresOptimized for one thread’s latencyGood at branching, varied codeLarge caches per coreRuns the OS, control logic, dataloadingGPU: thousands of simple unitsOptimized for total throughputSame operation over lots of dataVery high memory bandwidthRuns the matrix math of trainingand inferenceNeural networks are mostly big matrixmultiplications, which is exactly thework GPUs are built for.
Latency-optimized versus throughput-optimized.

03 · In the interview

  • Explaining why ML workloads run on GPUs or other accelerators.
  • Spotting CPU-side bottlenecks — data loading, preprocessing — that leave GPUs idle.
  • Understanding that a GPU server is still a computer with CPUs, memory, disks, and network that can fail.

04 · Practice question

“GPU utilization on our training job swings between 100% and near zero every few seconds. What would you look at?”

A strong answer covers

  1. The GPUs are probably waiting on input: data loading or preprocessing on the CPU can’t keep up.
  2. Check CPU usage of the data loader workers, storage or network read throughput, and whether batches are prefetched.
  3. Periodic stalls can also be checkpoint writes or synchronization between GPUs.
  4. Fixes: more loader workers, prefetching, caching data locally, faster formats, overlapping I/O with compute.