map / ai-hardware / failure-at-scale
AI Hardware · principle 7 of 7
Failure at cluster scale & checkpointing
A synchronous job needs every GPU healthy at once, so its reliability is p^N — and checkpointing is how you buy it back.
01 · Why it’s true
If each component works with probability p, and a job needs all N of them to keep working, the chance the job keeps working is pᴺ. Even when p is very close to 1, pᴺ drops quickly as N grows.
So in a synchronous job across thousands of GPUs, a failure somewhere becomes normal rather than rare. The system has to expect failures: checkpoint regularly, detect bad hardware quickly, replace it, and resume — the time lost to restarts becomes a capacity planning number.
02 · Picture it
03 · In the interview
- Explaining why large training jobs are designed around frequent failure.
- Choosing checkpoint intervals: frequent checkpoints cost time, rare ones lose more work per failure.
- Building fast detection and automatic replacement of bad nodes.
04 · Practice question
“Each GPU in a 4,000-GPU training job has a small chance of failing on any given day. Why is that a big problem, and how do you design around it?”
A strong answer covers
- The job needs every GPU, so its chance of a clean day is p⁴⁰⁰⁰ — tiny unless p is extremely close to 1; failures become a daily event.
- Checkpoint regularly so a failure only loses the work since the last checkpoint.
- Detect bad hardware fast (health checks, error counters), keep spare nodes, and automate the restart.
- Balance checkpoint frequency against its cost; track the share of time spent on useful training.
Want to play with it? Open the interactive deep dive →