map / ai-hardware / roofline / interactive
AI Hardware · interactive deep dive
The roofline model & failure at GPU scale
Two roofs. Your kernel hits one of them.
Every kernel does some math (FLOPs) on some data it has to move from memory (bytes). The ratio is its arithmetic intensity. The chip can deliver at most its peak compute, and at most bandwidth × intensity — whichever is lower:
attainable = min(peak FLOP/s, bandwidth × intensity)The ridge point — peak ÷ bandwidth — is the intensity you need to keep the math units busy. For modern accelerators it’s in the hundreds of FLOPs per byte. Most real workloads sit to the left of it.
Why GPUs sit idle during inference.
Generating one token with a large language model touches every weight once and does about 2 FLOPs with it (a multiply and an add) per sequence in the batch. With 16-bit weights, that’s 2 bytes moved per 2·B FLOPs — an intensity of roughly B, the batch size. At batch 1 the intensity is about 1, far left of a ridge in the hundreds, so the chip is waiting on memory almost all the time.
That one fact explains most inference engineering: batching requests together (raises FLOPs per byte), quantizing weights to fewer bytes (fewer bytes per FLOP), and why memory bandwidth — not peak FLOPs — is the spec that often predicts serving throughput. Try it: set LLM decode, then drag the batch size.
Simplification: this ignores the KV cache, which adds memory traffic that grows with sequence length and pushes long-context decode further toward the memory roof.
At scale, reliability is p^N.
Large-scale training is usually synchronous: every step waits for every GPU, so one failed or hung GPU stalls the whole job. That makes the job a chain of N dependencies in series. If each GPU gets through a day without an interruption with probability p, the whole job does so with probability:
P(job ok) = pN · mean time between interruptions ≈ MTBFgpu / NA per-GPU rate that would be invisible on a single machine becomes a several-times-a-day event at thousands of GPUs. The per-GPU rate below is an assumption you control — plug in your own fleet’s numbers.
Checkpointing: buying reliability back.
You can’t make p^N go away, so you make interruptions cheap. A checkpoint saves training state so a restart loses only the work since the last save. Checkpoint too often and you waste time writing; too rarely and each failure throws away more work. A classic first-order answer is Young’s approximation:
τopt ≈ √(2 · C · M)where C is the time to write a checkpoint and M is the mean time between interruptions. The widget above uses it to estimate the interval and the fraction of time lost to writing checkpoints, redoing lost work (on average half an interval per failure), and restarting.
The levers fall straight out of the formula: make C smaller (asynchronous or in-memory checkpoints), make restarts faster (fast failure detection, hot spares, quick reload), and make M larger (pre-flight health checks that catch bad GPUs before they join the job).
How it shows up in interviews.
Strong candidates ask whether the workload is memory-bound before blaming the scheduler, and reach for batching or kernel fusion.
The signal is recognizing a serial system: reliability falls as p^N, so the answer is cheap recovery, not perfect hardware.
Web services add replicas to raise availability. A synchronous job adds GPUs and lowers it. Saying that out loud shows you get the difference.
A sample question, and a strong answer dissected.
“This job is a serial system: every step needs every GPU. Job reliability is p^N, so 8× the GPUs means roughly 8× the interruptions at the same per-GPU rate. More interruptions are the expected outcome, not necessarily a regression.”
Naming the system shape first shows you reason from structure, not symptoms.
“I’d split it into two numbers: how often we’re interrupted, broken down by cause, and how much each interruption costs — detection time, restart time, and lost work since the last checkpoint.”
Separating rate from cost keeps the conversation grounded in data.
“Fewer interruptions: health-check nodes before they join the job, drain nodes with rising error counts, and burn in new hardware before it enters the pool.”
Shows practical knowledge of where failures come from in GPU fleets.
“Cheaper interruptions: detect hangs quickly with collective timeouts, keep hot spares so restarts don’t wait on repairs, and tune the checkpoint interval — roughly √(2·C·M) — while making checkpoint writes asynchronous so C shrinks.”
This is the lever that scales — and the one most candidates miss.
“I’d report goodput — the fraction of wall-clock time that turns into training progress — instead of uptime, and set a target for it. That’s the number leadership actually cares about.”
Ending on a goal-level metric aligns engineering work with what leadership cares about.