map / ai-hardware / hbm

AI Hardware · principle 3 of 7

HBM & the memory wall

Accelerator compute has grown faster than memory bandwidth, so feeding the chip — not doing the math — is often the bottleneck.

01 · Why it’s true

Large models are limited as much by memory as by compute. A model’s weights take parameters × bytes per parameter: a 70-billion-parameter model in 16-bit precision needs about 70 × 2 = 140 GB just for weights — more than a single GPU’s memory on many current accelerators.

GPUs use high-bandwidth memory (HBM) stacked next to the chip because they need to read data extremely fast. Its capacity is limited, so the model, the working memory for requests (such as the attention key-value cache), and for training the gradients and optimizer state all compete for it. Running out of GPU memory is one of the most common failures in ML work.

02 · Picture it

1Model weightsparams × byteseach+2KV cachegrows withcontext andbatch+3Optimizerstatetraining only≤ ?4GPU memorymust fit, orsplit / quantizeWeights alone: 70B parameters × 2 bytes (16-bit) ≈ 140 GB — more than one GPU holds onmany current accelerators.
1Model weightsparams × bytes each+2KV cachegrows with context and batch+3Optimizer statetraining only≤ ?4GPU memorymust fit, or split / quantizeWeights alone: 70B parameters × 2bytes (16-bit) ≈ 140 GB — more thanone GPU holds on many currentaccelerators.
Everything a model needs has to fit in the fast memory next to the GPU.

03 · In the interview

  • Back-of-envelope sizing: how many GPUs does a model need?
  • Diagnosing out-of-memory errors on GPUs.
  • Understanding why longer contexts and bigger batches need more memory.

04 · Practice question

“Roughly how much GPU memory do you need to serve a 70-billion-parameter model, and what else uses memory besides the weights?”

A strong answer covers

  1. Weights: 70B × 2 bytes (16-bit) ≈ 140 GB; with 8-bit weights about 70 GB.
  2. Plus the key-value cache, which grows with context length and the number of requests in flight.
  3. Plus runtime overhead and activations, so leave headroom.
  4. If one GPU can’t hold it, split the model across several GPUs (tensor or pipeline parallelism), or quantize to fewer bytes per parameter.