map / distributed-systems / saturation
Distributed Systems · principle 1 of 6
Saturation
As a resource gets busier, time spent waiting grows faster and faster — a system near 100% busy is already slow.
01 · Why it’s true
Requests don’t arrive evenly; they arrive in bursts. When a server is lightly used, a burst finds it idle and is handled right away. As it gets busier, a new request is more and more likely to arrive while others are still being served, so it has to wait in line.
Waiting doesn’t grow in a straight line with utilization. In simple queueing models, the average wait is proportional to u / (1 − u), where u is the fraction of time the server is busy: going from 50% to 90% busy multiplies the wait by nine. Near 100%, small increases in load cause huge increases in latency — so you plan capacity with headroom, not to 100%.
02 · Picture it
03 · In the interview
- Capacity planning: why you keep headroom and don’t target 100% utilization.
- Explaining why latency suddenly spikes after a small traffic increase, or after losing one node.
- Using saturation — queue length, run queue, connection pool waits — as an early warning signal.
04 · Practice question
“Our service runs at 85% CPU and latency is fine. Traffic grows 10%, and p99 latency triples. Why?”
A strong answer covers
- Near full utilization, waiting time rises steeply: 85% → about 94% busy lands on the steep part of the curve.
- Each request now usually arrives behind others, and bursts take longer to drain.
- The same thing happens when one of a few nodes fails and the rest absorb its load.
- Fix: add capacity or shed load now; long term, plan for enough headroom to absorb peaks and the loss of a node.