map / distributed-systems / retries-cascades
Distributed Systems · principle 6 of 6
Retries & cascades
Retries multiply load exactly when a system is weakest, so without budgets and backoff they turn a slowdown into an outage.
01 · Why it’s true
Retrying makes sense for a brief blip, but when a service is failing because it’s overloaded, each retry adds more load to the thing that’s already struggling.
Retries also multiply across layers. If each of three layers retries up to 3 times, one user request can become up to 3 × 3 × 3 = 27 attempts at the bottom layer. Overload in one service can then spread to everything that depends on it. Backoff with jitter, retry budgets, timeouts, and circuit breakers keep retries from turning a blip into an outage.
02 · Picture it
03 · In the interview
- Designing client behavior: timeouts, retries, backoff, and jitter.
- Explaining retry storms and the thundering herd problem.
- Using circuit breakers and load shedding to protect overloaded services.
04 · Practice question
“A database had a brief slowdown, but the whole system stayed down for 30 minutes after it recovered. What might have happened?”
A strong answer covers
- Clients timed out and retried, often at several layers, multiplying load beyond normal.
- The database never got the chance to recover; synchronized retries also arrive in waves.
- Mitigations: exponential backoff with jitter, retries in only one layer, retry budgets, and circuit breakers.
- Add load shedding so the database rejects excess work quickly instead of slowing down for everyone.