map / distributed-systems / saturation

Distributed Systems · principle 1 of 6

Saturation

As a resource gets busier, time spent waiting grows faster and faster — a system near 100% busy is already slow.

01 · Why it’s true

Requests don’t arrive evenly; they arrive in bursts. When a server is lightly used, a burst finds it idle and is handled right away. As it gets busier, a new request is more and more likely to arrive while others are still being served, so it has to wait in line.

Waiting doesn’t grow in a straight line with utilization. In simple queueing models, the average wait is proportional to u / (1 − u), where u is the fraction of time the server is busy: going from 50% to 90% busy multiplies the wait by nine. Near 100%, small increases in load cause huge increases in latency — so you plan capacity with headroom, not to 100%.

02 · Picture it

↑ average time waitinghow busy the resource is →0%50%70%90%100%50% busy: wait ≈ 1×90% busy: wait ≈ 9×headroom: short waitsnear 100%: waits explode →
↑ average time waitinghow busy the resource is →0%50%90%50%: wait ≈ 1×90%: wait ≈ 9×Green: headroom, short waits. Red:near 100% busy, waits explode.
Waiting time is flat with headroom, then climbs steeply near 100% busy.

03 · In the interview

  • Capacity planning: why you keep headroom and don’t target 100% utilization.
  • Explaining why latency suddenly spikes after a small traffic increase, or after losing one node.
  • Using saturation — queue length, run queue, connection pool waits — as an early warning signal.

04 · Practice question

“Our service runs at 85% CPU and latency is fine. Traffic grows 10%, and p99 latency triples. Why?”

A strong answer covers

  1. Near full utilization, waiting time rises steeply: 85% → about 94% busy lands on the steep part of the curve.
  2. Each request now usually arrives behind others, and bursts take longer to drain.
  3. The same thing happens when one of a few nodes fails and the rest absorb its load.
  4. Fix: add capacity or shed load now; long term, plan for enough headroom to absorb peaks and the loss of a node.