map / ai-hardware

Pillar 05 · 7 principles

AI Hardware.

Reliability at the scale of thousands of GPUs.

AI infrastructure changes the reliability math. A training job is one giant synchronous system, and accelerator performance is usually limited by memory bandwidth, not FLOPs. Few prep resources cover this — so we do.

01

The roofline model

A kernel’s speed is capped by either compute or memory bandwidth, and its arithmetic intensity (FLOPs per byte) tells you which.

An interactive roofline: pick hardware and a workload, see which roof you hit.

Interactive →
02

GPU vs CPU

CPUs spend silicon on making one thread fast; GPUs spend it on running thousands of simple threads at once.

A few large cores vs many small cores working through the same array.

Coming soon
03

HBM & the memory wall

Accelerator compute has grown faster than memory bandwidth, so feeding the chip — not doing the math — is often the bottleneck.

Compute units waiting on data streaming in from stacked memory.

Coming soon
04

NVLink & InfiniBand

Within a node GPUs talk over fast local links; across nodes they go over the network, and the slowest link sets the pace of a collective.

A cluster topology with intra-node and inter-node bandwidth tiers.

Coming soon
05

Training vs inference

Training is a long, synchronous, throughput-bound batch job; inference is a latency-bound service — they fail and scale differently.

One job spanning the cluster vs many independent replicas serving requests.

Coming soon
06

Data, tensor & pipeline parallelism

You can split the data, split each layer, or split the stack of layers — each choice trades memory for communication.

The same model sliced three ways across GPUs.

Coming soon
07

Failure at cluster scale & checkpointing

A synchronous job needs every GPU healthy at once, so its reliability is p^N — and checkpointing is how you buy it back.

Grow the cluster and watch job-level availability and time-between-interruptions fall.

Interactive →

How it shows up in interviews

  • “Our GPUs show low utilization — why, and what would you do?”
  • “A large training job keeps getting interrupted — how do you think about it?”
  • Checkpointing strategy, health checking, and blast radius in GPU clusters.

Next pillar

01 · Networking

Every request is a chain of round trips.

Go →