map / ai-hardware / hbm
AI Hardware · principle 3 of 7
HBM & the memory wall
Accelerator compute has grown faster than memory bandwidth, so feeding the chip — not doing the math — is often the bottleneck.
01 · Why it’s true
Large models are limited as much by memory as by compute. A model’s weights take parameters × bytes per parameter: a 70-billion-parameter model in 16-bit precision needs about 70 × 2 = 140 GB just for weights — more than a single GPU’s memory on many current accelerators.
GPUs use high-bandwidth memory (HBM) stacked next to the chip because they need to read data extremely fast. Its capacity is limited, so the model, the working memory for requests (such as the attention key-value cache), and for training the gradients and optimizer state all compete for it. Running out of GPU memory is one of the most common failures in ML work.
02 · Picture it
03 · In the interview
- Back-of-envelope sizing: how many GPUs does a model need?
- Diagnosing out-of-memory errors on GPUs.
- Understanding why longer contexts and bigger batches need more memory.
04 · Practice question
“Roughly how much GPU memory do you need to serve a 70-billion-parameter model, and what else uses memory besides the weights?”
A strong answer covers
- Weights: 70B × 2 bytes (16-bit) ≈ 140 GB; with 8-bit weights about 70 GB.
- Plus the key-value cache, which grows with context length and the number of requests in flight.
- Plus runtime overhead and activations, so leave headroom.
- If one GPU can’t hold it, split the model across several GPUs (tensor or pipeline parallelism), or quantize to fewer bytes per parameter.