map / distributed-systems / queueing
Distributed Systems · flagship principle
Queueing & Little’s Law
Drag utilization. Watch latency explode.
Model a server as one queue: requests arrive at rate λ, each takes on average S to serve. The server is busy a fraction ρ = λ·S of the time. For the classic M/M/1 queue (random arrivals, random service times, one server), mean time in the system is:
W = S / (1 − ρ)At 50% utilization, a request takes 2× its service time on average. At 80%, 5×. At 90%, 10×. At 95%, 20×. The last 10 points of utilization cost more than the first 90. That’s why well-run services target utilization well below 100% — headroom isn’t waste, it’s what keeps latency flat.
Tails are worse than means. In M/M/1, time in the system is exponentially distributed, so p99 is about ln(100) ≈ 4.6× the mean. When the mean doubles, so does the p99.
Little’s Law: the one equation you can always use.
Little’s Law holds for any stable system, regardless of arrival pattern or service distribution: the average number of items inside equals the arrival rate times the average time each spends inside.
L = λ · WIt turns latency into a capacity number. 2,000 req/s at 50 ms means 100 requests in flight, on average — so 100 connections, threads, or memory buffers held at once. If a downstream dependency slows from 50 ms to 500 ms, the same traffic needs 1,000 in flight.
Where it bites: losing a node.
Take N identical nodes behind a load balancer, each at utilization ρ. Lose one, and the same traffic is spread over N − 1 nodes: each now runs at ρ·N / (N − 1). Because latency is nonlinear in ρ, a modest bump in utilization can be a large bump in latency.
Modeling note: each node is treated as its own M/M/1 queue. Real servers process many requests concurrently (closer to M/M/c), which keeps latency flatter at moderate utilization but makes the knee sharper near saturation. The shape of the argument holds either way.
How it shows up in interviews.
“How many servers do we need?” Interviewers want you to size for a target utilization and latency, not for 100% of peak.
“Traffic went up 15% and p99 tripled — why?” The nonlinearity of W = S/(1−ρ) is the answer they’re fishing for.
“How big should this thread or connection pool be?” Little’s Law gives you a defensible number in one line.
A sample question, and a strong answer dissected.
“First I’d confirm CPU is the actual bottleneck resource and that 85% is measured at peak, not as a daily average — and what our latency SLO is.”
Shows you know CPU% is a proxy, and that the bottleneck might be elsewhere.
“If load spreads evenly, each remaining server goes to 85% × 10/9 ≈ 94%. Queueing delay is nonlinear: in a simple single-queue model, mean latency scales with 1/(1−ρ).”
Reaching for a model instead of a gut feeling is the senior signal.
“That’s 1/0.15 ≈ 6.7× service time going to 1/0.056 ≈ 18× — roughly 2.7× worse mean latency at peak, and the tail moves with it.”
A number makes the risk concrete for a non-engineering stakeholder.
“It’s worse during an incident: at 9 servers, losing one more puts each at 85% × 10/8 ≈ 106% — demand exceeds capacity and queues grow without bound. Today we can survive a node loss; after the cut we can’t.”
Thinking about N+1 shows you design for failure, not for the happy path.
“If cost matters, there are better levers: cut service time with profiling, autoscale so off-peak runs lean, or right-size instance types. I’d run a load test at the target utilization before deciding.”
Ending with options, not a flat ‘no’, is what makes this a strong answer.