map / ai-hardware / failure-at-scale

AI Hardware · principle 7 of 7

Failure at cluster scale & checkpointing

A synchronous job needs every GPU healthy at once, so its reliability is p^N — and checkpointing is how you buy it back.

01 · Why it’s true

If each component works with probability p, and a job needs all N of them to keep working, the chance the job keeps working is pᴺ. Even when p is very close to 1, pᴺ drops quickly as N grows.

So in a synchronous job across thousands of GPUs, a failure somewhere becomes normal rather than rare. The system has to expect failures: checkpoint regularly, detect bad hardware quickly, replace it, and resume — the time lost to restarts becomes a capacity planning number.

02 · Picture it

↑ chance the whole job runs cleanGPUs in one synchronous job (log scale) →1101001,00010,000100 GPUs: ≈ 90%1,000: ≈ 37%4,000: ≈ 2%Illustration: each GPU has a 99.9% chance of getting through the run. The job needs allof them, so its odds are 0.999ᴺ.
↑ chance the whole job runs cleanGPUs in one job (log) →110010,000100 GPUs: ≈ 90%1,000: ≈ 37%4,000: ≈ 2%Illustration: each GPU has a 99.9%chance of getting through the run. Thejob needs all of them, so its odds are0.999ᴺ.
The chance that all N parts keep working falls fast as N grows.

03 · In the interview

  • Explaining why large training jobs are designed around frequent failure.
  • Choosing checkpoint intervals: frequent checkpoints cost time, rare ones lose more work per failure.
  • Building fast detection and automatic replacement of bad nodes.

04 · Practice question

“Each GPU in a 4,000-GPU training job has a small chance of failing on any given day. Why is that a big problem, and how do you design around it?”

A strong answer covers

  1. The job needs every GPU, so its chance of a clean day is p⁴⁰⁰⁰ — tiny unless p is extremely close to 1; failures become a daily event.
  2. Checkpoint regularly so a failure only loses the work since the last checkpoint.
  3. Detect bad hardware fast (health checks, error counters), keep spare nodes, and automate the restart.
  4. Balance checkpoint frequency against its cost; track the share of time spent on useful training.

Want to play with it? Open the interactive deep dive →