map / ai-hardware / parallelism

AI Hardware · principle 6 of 7

Data, tensor & pipeline parallelism

You can split the data, split each layer, or split the stack of layers — each choice trades memory for communication.

01 · Why it’s true

When a model or its training data is too big for one GPU, you split the work. Data parallelism gives every GPU a full copy of the model and different data, then averages gradients. Tensor parallelism splits each layer’s matrices across GPUs. Pipeline parallelism gives different layers to different GPUs.

Each split creates communication. Tensor parallelism communicates inside every layer, so it usually stays within a server on the fastest links; data parallelism communicates once per step, so it can span servers. Real systems combine them to fit memory and keep communication on the right links.

02 · Picture it

STRATEGYWHAT’S SPLITCOMMUNICATIONUSUALLY PLACEDData parallelthe data; each GPUhas the full modelaverage gradientseach stepacross serversTensor paralleleach layer’smatricesinside every layer —heavywithin one serverPipelineparallelgroups of layersactivations betweenstagesacross servers
Data parallelWHAT’S SPLITthe data; each GPU has the full modelCOMMUNICATIONaverage gradients each stepUSUALLY PLACEDacross serversTensor parallelWHAT’S SPLITeach layer’s matricesCOMMUNICATIONinside every layer — heavyUSUALLY PLACEDwithin one serverPipeline parallelWHAT’S SPLITgroups of layersCOMMUNICATIONactivations between stagesUSUALLY PLACEDacross servers
Three ways to split a model, and what each costs in communication.

03 · In the interview

  • Explaining why a model is sharded the way it is.
  • Matching communication patterns to network topology.
  • Understanding how one failure affects jobs that use each strategy.

04 · Practice question

“A model doesn’t fit in one GPU’s memory. What are your options, and what do they cost?”

A strong answer covers

  1. Tensor parallelism: split layers across GPUs in one server; heavy communication, so it needs the fastest links.
  2. Pipeline parallelism: split layers into stages across GPUs; less communication, but idle time unless you keep several micro-batches in flight.
  3. Reduce memory: lower precision, quantization, or sharding the optimizer state.
  4. Data parallelism alone doesn’t help it fit — every GPU still holds the full model — but it adds throughput on top.