map / distributed-systems / retries-cascades

Distributed Systems · principle 6 of 6

Retries & cascades

Retries multiply load exactly when a system is weakest, so without budgets and backoff they turn a slowdown into an outage.

01 · Why it’s true

Retrying makes sense for a brief blip, but when a service is failing because it’s overloaded, each retry adds more load to the thing that’s already struggling.

Retries also multiply across layers. If each of three layers retries up to 3 times, one user request can become up to 3 × 3 × 3 = 27 attempts at the bottom layer. Overload in one service can then spread to everything that depends on it. Backoff with jitter, retry budgets, timeouts, and circuit breakers keep retries from turning a blip into an outage.

02 · Picture it

1User1 request12APIretries up to 3×≤ 33Serviceeach callretries up to 3×≤ 94Databaseup to 9 callsRetries multiply at each layer and hit hardest when a service is already struggling.Retry in one layer, back off with jitter, and cap retries as a share of traffic.
1User1 request12APIretries up to 3×≤ 33Serviceeach call retries up to 3×≤ 94Databaseup to 9 callsRetries multiply at each layer and hithardest when a service is alreadystruggling. Retry in one layer, backoff with jitter, and cap retries as ashare of traffic.
One user request can become many at the bottom of the stack.

03 · In the interview

  • Designing client behavior: timeouts, retries, backoff, and jitter.
  • Explaining retry storms and the thundering herd problem.
  • Using circuit breakers and load shedding to protect overloaded services.

04 · Practice question

“A database had a brief slowdown, but the whole system stayed down for 30 minutes after it recovered. What might have happened?”

A strong answer covers

  1. Clients timed out and retried, often at several layers, multiplying load beyond normal.
  2. The database never got the chance to recover; synchronized retries also arrive in waves.
  3. Mitigations: exponential backoff with jitter, retries in only one layer, retry budgets, and circuit breakers.
  4. Add load shedding so the database rejects excess work quickly instead of slowing down for everyone.