map / ai-hardware
Pillar 05 · 7 principles
AI Hardware.
Reliability at the scale of thousands of GPUs.
AI infrastructure changes the reliability math. A training job is one giant synchronous system, and accelerator performance is usually limited by memory bandwidth, not FLOPs. Few prep resources cover this — so we do.
The roofline model
A kernel’s speed is capped by either compute or memory bandwidth, and its arithmetic intensity (FLOPs per byte) tells you which.
An interactive roofline: pick hardware and a workload, see which roof you hit.
GPU vs CPU
CPUs spend silicon on making one thread fast; GPUs spend it on running thousands of simple threads at once.
A few large cores vs many small cores working through the same array.
HBM & the memory wall
Accelerator compute has grown faster than memory bandwidth, so feeding the chip — not doing the math — is often the bottleneck.
Compute units waiting on data streaming in from stacked memory.
NVLink & InfiniBand
Within a node GPUs talk over fast local links; across nodes they go over the network, and the slowest link sets the pace of a collective.
A cluster topology with intra-node and inter-node bandwidth tiers.
Training vs inference
Training is a long, synchronous, throughput-bound batch job; inference is a latency-bound service — they fail and scale differently.
One job spanning the cluster vs many independent replicas serving requests.
Data, tensor & pipeline parallelism
You can split the data, split each layer, or split the stack of layers — each choice trades memory for communication.
The same model sliced three ways across GPUs.
Failure at cluster scale & checkpointing
A synchronous job needs every GPU healthy at once, so its reliability is p^N — and checkpointing is how you buy it back.
Grow the cluster and watch job-level availability and time-between-interruptions fall.
How it shows up in interviews
- “Our GPUs show low utilization — why, and what would you do?”
- “A large training job keeps getting interrupted — how do you think about it?”
- Checkpointing strategy, health checking, and blast radius in GPU clusters.