map / distributed-systems / availability-math / interactive
Distributed Systems · interactive deep dive
Availability math & error budgets
Compose a system. Watch the nines drain.
Every request here must pass through every component in the chain. Change a component’s availability, add dependencies, or give one a redundant copy, and watch what happens to the whole.
serial: A = ∏ Aᵢ · parallel: A = 1 − ∏ (1 − Aᵢ)Two things jump out. Adding hard dependencies always makes the system less available than its weakest part. And redundancy is powerful: two independent 99.9% copies give 99.9999% in theory — the catch is the word independent. Shared power, a shared config push, or a shared bug fails both copies together.
What the nines actually buy you.
Allowed downtime is just (1 − availability) × the window. Worth knowing a few by heart:
Each extra nine is 10× less downtime — and usually far more than 10× the cost. At 99.99%, a single incident that takes five minutes to page a human has already blown the month.
Error budgets: spend unreliability on purpose.
An SLO of 99.9% means 0.1% of requests may fail — that’s the error budget. Teams spend it on risk: deploys, experiments, migrations. The burn rate says how fast you’re spending it relative to plan: a burn rate of 1 uses exactly the whole budget over the window.
burn rate = error rate ÷ (1 − SLO)How it shows up in interviews.
“What availability can this design achieve?” Multiply the hard dependencies and name the weakest link.
“How would you set and alert on an SLO?” Error budgets and multi-window burn-rate alerts are the expected vocabulary.
“Is a fourth nine worth it?” Translate nines into minutes, then weigh cost against what users actually notice.
A sample question, and a strong answer dissected.
“If all five are hard dependencies in series, checkout’s ceiling is 0.999⁵ ≈ 99.5% — about 3.6 hours of allowed failure per 30 days, versus about 4.3 minutes for 99.99%. That’s before checkout’s own failures.”
Doing the arithmetic out loud anchors the whole discussion.
“To hit 99.99% with five serial dependencies, each would need roughly 99.998% — that’s the fifth root of 0.9999. Asking five teams for that is far harder than changing the design.”
Working backwards from the target shows the requirement is unreasonable as designed.
“Make dependencies soft where possible: degrade gracefully if recommendations or loyalty points are down, move emails and analytics to an async queue, and cache data that can tolerate staleness. Every dependency removed from the critical path multiplies availability back up.”
The highest-leverage fix is architectural, not operational.
“For what must stay synchronous, run redundant instances in independent failure domains with tested failover — and add timeouts so a slow dependency fails fast instead of hanging checkout.”
Qualifying ‘independent’ shows you understand where redundancy math breaks.
“Finally I’d ask what users need. If 99.95% measured at the checkout API meets the business need, the error budget gives us room to keep shipping.”
Tying the SLO to user experience is what makes it a good SLO.