map / ai-hardware / parallelism
AI Hardware · principle 6 of 7
Data, tensor & pipeline parallelism
You can split the data, split each layer, or split the stack of layers — each choice trades memory for communication.
01 · Why it’s true
When a model or its training data is too big for one GPU, you split the work. Data parallelism gives every GPU a full copy of the model and different data, then averages gradients. Tensor parallelism splits each layer’s matrices across GPUs. Pipeline parallelism gives different layers to different GPUs.
Each split creates communication. Tensor parallelism communicates inside every layer, so it usually stays within a server on the fastest links; data parallelism communicates once per step, so it can span servers. Real systems combine them to fit memory and keep communication on the right links.
02 · Picture it
03 · In the interview
- Explaining why a model is sharded the way it is.
- Matching communication patterns to network topology.
- Understanding how one failure affects jobs that use each strategy.
04 · Practice question
“A model doesn’t fit in one GPU’s memory. What are your options, and what do they cost?”
A strong answer covers
- Tensor parallelism: split layers across GPUs in one server; heavy communication, so it needs the fastest links.
- Pipeline parallelism: split layers into stages across GPUs; less communication, but idle time unless you keep several micro-batches in flight.
- Reduce memory: lower precision, quantization, or sharding the optimizer state.
- Data parallelism alone doesn’t help it fit — every GPU still holds the full model — but it adds throughput on top.