map / ai-hardware / roofline / interactive

AI Hardware · interactive deep dive

The roofline model & failure at GPU scale

The principleA chip can only go as fast as its slowest supply line: a kernel is capped by either compute or memory bandwidth, and its FLOPs-per-byte tells you which. At cluster scale the same logic applies to reliability — a synchronous job is only as healthy as its least healthy GPU.
01

Two roofs. Your kernel hits one of them.

Every kernel does some math (FLOPs) on some data it has to move from memory (bytes). The ratio is its arithmetic intensity. The chip can deliver at most its peak compute, and at most bandwidth × intensity — whichever is lower:

attainable = min(peak FLOP/s, bandwidth × intensity)

The ridge point — peak ÷ bandwidth — is the intensity you need to keep the math units busy. For modern accelerators it’s in the hundreds of FLOPs per byte. Most real workloads sit to the left of it.

Roofline · attainable = min(peak, bandwidth × intensity)
0.11101001,00010,0000.11101001,000arithmetic intensity (FLOP / byte, log)attainable TFLOP/s (log)ridge ≈ 295compute roof · 989 TFLOP/smemory roof · 3.35 TB/smemory-bound region
intensity
1
ridge point
295
attainable
3.35 TF/s
% of peak
0.34%
Memory-bound. Each decode step streams every weight from HBM to do ~2 FLOPs per weight per sequence. Batch more sequences, or store weights in fewer bytes, to move right.

Hardware presets are approximate datasheet peaks (dense BF16, HBM). Intensities are idealized: each byte moved once.

02

Why GPUs sit idle during inference.

Generating one token with a large language model touches every weight once and does about 2 FLOPs with it (a multiply and an add) per sequence in the batch. With 16-bit weights, that’s 2 bytes moved per 2·B FLOPs — an intensity of roughly B, the batch size. At batch 1 the intensity is about 1, far left of a ridge in the hundreds, so the chip is waiting on memory almost all the time.

That one fact explains most inference engineering: batching requests together (raises FLOPs per byte), quantizing weights to fewer bytes (fewer bytes per FLOP), and why memory bandwidth — not peak FLOPs — is the spec that often predicts serving throughput. Try it: set LLM decode, then drag the batch size.

Simplification: this ignores the KV cache, which adds memory traffic that grows with sequence length and pushes long-context decode further toward the memory roof.

03

At scale, reliability is p^N.

Large-scale training is usually synchronous: every step waits for every GPU, so one failed or hung GPU stalls the whole job. That makes the job a chain of N dependencies in series. If each GPU gets through a day without an interruption with probability p, the whole job does so with probability:

P(job ok) = pN  ·  mean time between interruptions ≈ MTBFgpu / N

A per-GPU rate that would be invisible on a single machine becomes a several-times-a-day event at thousands of GPUs. The per-GPU rate below is an assumption you control — plug in your own fleet’s numbers.

Synchronous job · every GPU must be healthy

Each square = 4 GPUs. Simulated day: 6 interruptions. Each one stalls all 4,096 GPUs until the job restarts.

0%50%100%8645124,09632,768P(no interruption in 24h) = (1 − f)^N
P(clean day)
1.7%
interrupts / day
4.1
mean time between
5.9 h
ckpt every τ
59 min
≈ time lost to checkpoints + rework + restarts
21.1%
04

Checkpointing: buying reliability back.

You can’t make p^N go away, so you make interruptions cheap. A checkpoint saves training state so a restart loses only the work since the last save. Checkpoint too often and you waste time writing; too rarely and each failure throws away more work. A classic first-order answer is Young’s approximation:

τopt ≈ √(2 · C · M)

where C is the time to write a checkpoint and M is the mean time between interruptions. The widget above uses it to estimate the interval and the fraction of time lost to writing checkpoints, redoing lost work (on average half an interval per failure), and restarting.

The levers fall straight out of the formula: make C smaller (asynchronous or in-memory checkpoints), make restarts faster (fast failure detection, hot spares, quick reload), and make M larger (pre-flight health checks that catch bad GPUs before they join the job).

05

How it shows up in interviews.

“Low GPU utilization”

Strong candidates ask whether the workload is memory-bound before blaming the scheduler, and reach for batching or kernel fusion.

“The job keeps dying”

The signal is recognizing a serial system: reliability falls as p^N, so the answer is cheap recovery, not perfect hardware.

Web SRE vs AI SRE

Web services add replicas to raise availability. A synchronous job adds GPUs and lowers it. Saying that out loud shows you get the difference.

06

A sample question, and a strong answer dissected.

The question“We’re scaling a training job from 512 to 4,096 GPUs, and it’s getting interrupted far more often. Leadership asks why we can’t just make it reliable. How do you approach it?”
Reframe

“This job is a serial system: every step needs every GPU. Job reliability is p^N, so 8× the GPUs means roughly 8× the interruptions at the same per-GPU rate. More interruptions are the expected outcome, not necessarily a regression.”

Naming the system shape first shows you reason from structure, not symptoms.

Measure

“I’d split it into two numbers: how often we’re interrupted, broken down by cause, and how much each interruption costs — detection time, restart time, and lost work since the last checkpoint.”

Separating rate from cost keeps the conversation grounded in data.

Reduce rate

“Fewer interruptions: health-check nodes before they join the job, drain nodes with rising error counts, and burn in new hardware before it enters the pool.”

Shows practical knowledge of where failures come from in GPU fleets.

Reduce cost

“Cheaper interruptions: detect hangs quickly with collective timeouts, keep hot spares so restarts don’t wait on repairs, and tune the checkpoint interval — roughly √(2·C·M) — while making checkpoint writes asynchronous so C shrinks.”

This is the lever that scales — and the one most candidates miss.

Metric

“I’d report goodput — the fraction of wall-clock time that turns into training progress — instead of uptime, and set a target for it. That’s the number leadership actually cares about.”

Ending on a goal-level metric aligns engineering work with what leadership cares about.

07

Common pitfalls.

Peak FLOPs worshipComparing accelerators on peak FLOPs alone when the workload is memory-bound.
Web-service instincts“Add more replicas” raises availability for stateless services but lowers it for a synchronous job.
Fixed checkpoint cadenceAn interval tuned for 512 GPUs is wrong at 4,096: M shrank, so τ should too.
Only counting crashesHangs and slow GPUs stall a synchronous job just as hard as crashes, and are slower to detect.

Back to the principle

Roofline — the principle

Read →