map / distributed-systems / availability-math / interactive

Distributed Systems · interactive deep dive

Availability math & error budgets

The principleHard dependencies in series multiply their availabilities down; independent redundant copies in parallel multiply their failure probabilities down — so a system is bounded by its chain of hard dependencies, and redundancy only helps when failures are truly independent.
01

Compose a system. Watch the nines drain.

Every request here must pass through every component in the chain. Change a component’s availability, add dependencies, or give one a redundant copy, and watch what happens to the whole.

serial: A = ∏ Aᵢ   ·   parallel: A = 1 − ∏ (1 − Aᵢ)
Serial chain · parallel replicas
userload balancer99.99%99.99%app server99.9%99.9%database99.95%99.95%
componentavailabilityreplicas
system availability
99.84006%
nines
2.80
weakest link
app server
downtime / 30 days
69.1 min
downtime / year
14.0 h
Serial: multiply availabilities — A = A₁ · A₂ · …. Parallel copies: multiply failure probabilities — A = 1 − (1 − a)ⁿ. The chain can never be better than its weakest link.

Assumes failures are independent and failover is instant. Shared dependencies, correlated failures, and failover bugs make real redundancy worth less than this math says.

Two things jump out. Adding hard dependencies always makes the system less available than its weakest part. And redundancy is powerful: two independent 99.9% copies give 99.9999% in theory — the catch is the word independent. Shared power, a shared config push, or a shared bug fails both copies together.

02

What the nines actually buy you.

Allowed downtime is just (1 − availability) × the window. Worth knowing a few by heart:

availabilityper 30 daysper year
99%7.2 h3.65 days
99.9%43.2 min8.76 h
99.95%21.6 min4.38 h
99.99%4.32 min52.6 min
99.999%26 s5.26 min

Each extra nine is 10× less downtime — and usually far more than 10× the cost. At 99.99%, a single incident that takes five minutes to page a human has already blown the month.

03

Error budgets: spend unreliability on purpose.

An SLO of 99.9% means 0.1% of requests may fail — that’s the error budget. Teams spend it on risk: deploys, experiments, migrations. The burn rate says how fast you’re spending it relative to plan: a burn rate of 1 uses exactly the whole budget over the window.

burn rate = error rate ÷ (1 − SLO)
Error budget · 30-day window
0%50%100%day 0day 7day 14day 21day 30burn rate 1 = exactly on budget
budget
0.10%
≈ full outage time
43.2 min
burn rate
5.0×
budget gone in
6.0 days
Burn rate = error rate ÷ (1 − SLO). At 5.0×, a full 30-day budget is gone in 6.0 days. Alerting on burn rate (fast and slow windows) beats alerting on raw error rate.
04

How it shows up in interviews.

System design

“What availability can this design achieve?” Multiply the hard dependencies and name the weakest link.

SLOs

“How would you set and alert on an SLO?” Error budgets and multi-window burn-rate alerts are the expected vocabulary.

Trade-offs

“Is a fourth nine worth it?” Translate nines into minutes, then weigh cost against what users actually notice.

05

A sample question, and a strong answer dissected.

The question“Checkout synchronously calls five internal services, each with a 99.9% SLO. The product team wants checkout at 99.99%. Is that achievable? What would you change?”
Compute the ceiling

“If all five are hard dependencies in series, checkout’s ceiling is 0.999⁵ ≈ 99.5% — about 3.6 hours of allowed failure per 30 days, versus about 4.3 minutes for 99.99%. That’s before checkout’s own failures.”

Doing the arithmetic out loud anchors the whole discussion.

Invert it

“To hit 99.99% with five serial dependencies, each would need roughly 99.998% — that’s the fifth root of 0.9999. Asking five teams for that is far harder than changing the design.”

Working backwards from the target shows the requirement is unreasonable as designed.

Cut hard dependencies

“Make dependencies soft where possible: degrade gracefully if recommendations or loyalty points are down, move emails and analytics to an async queue, and cache data that can tolerate staleness. Every dependency removed from the critical path multiplies availability back up.”

The highest-leverage fix is architectural, not operational.

Add real redundancy

“For what must stay synchronous, run redundant instances in independent failure domains with tested failover — and add timeouts so a slow dependency fails fast instead of hanging checkout.”

Qualifying ‘independent’ shows you understand where redundancy math breaks.

Agree on the SLO

“Finally I’d ask what users need. If 99.95% measured at the checkout API meets the business need, the error budget gives us room to keep shipping.”

Tying the SLO to user experience is what makes it a good SLO.

06

Common pitfalls.

Averaging, not multiplyingFive 99.9% services in series are not “99.9% overall”.
Assuming independenceReplicas sharing a region, deploy pipeline, or config fail together.
Ignoring failoverRedundancy only counts if failover is fast, automatic, and tested.
Alerting on raw errorsA fixed error-rate threshold either pages too much or misses slow burns; alert on burn rate.

Back to the principle

Availability math — the principle

Read →