map / distributed-systems / queueing

Distributed Systems · flagship principle

Queueing & Little’s Law

The principleLatency doesn’t grow linearly with load. In a single queue, mean latency is the service time divided by the fraction of time the server is idle — so as utilization approaches 100%, wait time grows without bound.
01

Drag utilization. Watch latency explode.

Model a server as one queue: requests arrive at rate λ, each takes on average S to serve. The server is busy a fraction ρ = λ·S of the time. For the classic M/M/1 queue (random arrivals, random service times, one server), mean time in the system is:

W = S / (1 − ρ)
Single-server queue · M/M/1
05×10×15×20×0%20%40%60%80%100%ρ → 1: W → ∞utilization ρ = λ·Smean latency (× service time)
throughput λ
70/s
service S
10 ms
mean latency W
33.3 ms
queue wait
23.3 ms
p99 (≈4.6·W)
154 ms
in system L
2.33
From here, adding 5 points of utilization multiplies mean latency by 1.20×. Still on the flat part of the curve.

Little’s Law check: L = λ·W = 70/s × 33.3 ms = 2.33 ✓

At 50% utilization, a request takes 2× its service time on average. At 80%, 5×. At 90%, 10×. At 95%, 20×. The last 10 points of utilization cost more than the first 90. That’s why well-run services target utilization well below 100% — headroom isn’t waste, it’s what keeps latency flat.

Tails are worse than means. In M/M/1, time in the system is exponentially distributed, so p99 is about ln(100) ≈ 4.6× the mean. When the mean doubles, so does the p99.

02

Little’s Law: the one equation you can always use.

Little’s Law holds for any stable system, regardless of arrival pattern or service distribution: the average number of items inside equals the arrival rate times the average time each spends inside.

L = λ · W

It turns latency into a capacity number. 2,000 req/s at 50 ms means 100 requests in flight, on average — so 100 connections, threads, or memory buffers held at once. If a downstream dependency slows from 50 ms to 500 ms, the same traffic needs 1,000 in flight.

Little’s Law · sizing a worker pool
requests in flight L = λ·W
100
pool’s max throughput = pool / W
1280 req/s
You need ~100 concurrent workers but only have 64. Requests queue, latency grows, and that growth raises L further — this is how a slow dependency exhausts a thread pool.
03

Where it bites: losing a node.

Take N identical nodes behind a load balancer, each at utilization ρ. Lose one, and the same traffic is spread over N − 1 nodes: each now runs at ρ·N / (N − 1). Because latency is nonlinear in ρ, a modest bump in utilization can be a large bump in latency.

Lose one node · load redistributes

Before: 10 nodes at 80%

After: 9 nodes at 88.9%

new utilization ρ·N/(N−1)
88.9%
mean latency change (M/M/1)
1.80×
Survivable: you have enough headroom to lose a node.

The safe ceiling to survive one loss is ρ < (N−1)/N in pure capacity terms, and well below it once you care about latency.

Modeling note: each node is treated as its own M/M/1 queue. Real servers process many requests concurrently (closer to M/M/c), which keeps latency flatter at moderate utilization but makes the knee sharper near saturation. The shape of the argument holds either way.

04

How it shows up in interviews.

Capacity planning

“How many servers do we need?” Interviewers want you to size for a target utilization and latency, not for 100% of peak.

Latency mysteries

“Traffic went up 15% and p99 tripled — why?” The nonlinearity of W = S/(1−ρ) is the answer they’re fishing for.

Pools & limits

“How big should this thread or connection pool be?” Little’s Law gives you a defensible number in one line.

05

A sample question, and a strong answer dissected.

The question“Our API runs on 10 servers at about 85% CPU at peak, and p99 latency is fine. Finance wants to remove one server to save money. What do you tell them?”
Clarify

“First I’d confirm CPU is the actual bottleneck resource and that 85% is measured at peak, not as a daily average — and what our latency SLO is.”

Shows you know CPU% is a proxy, and that the bottleneck might be elsewhere.

Model

“If load spreads evenly, each remaining server goes to 85% × 10/9 ≈ 94%. Queueing delay is nonlinear: in a simple single-queue model, mean latency scales with 1/(1−ρ).”

Reaching for a model instead of a gut feeling is the senior signal.

Quantify

“That’s 1/0.15 ≈ 6.7× service time going to 1/0.056 ≈ 18× — roughly 2.7× worse mean latency at peak, and the tail moves with it.”

A number makes the risk concrete for a non-engineering stakeholder.

Failure mode

“It’s worse during an incident: at 9 servers, losing one more puts each at 85% × 10/8 ≈ 106% — demand exceeds capacity and queues grow without bound. Today we can survive a node loss; after the cut we can’t.”

Thinking about N+1 shows you design for failure, not for the happy path.

Options

“If cost matters, there are better levers: cut service time with profiling, autoscale so off-peak runs lean, or right-size instance types. I’d run a load test at the target utilization before deciding.”

Ending with options, not a flat ‘no’, is what makes this a strong answer.

06

Common pitfalls.

Linear thinking“We’re at 85%, so we have 15% headroom.” The headroom in latency terms is far smaller.
Averages over peaksSizing from daily average utilization hides the peak where the queue actually forms.
One resource onlyCPU isn’t the only queue: locks, connection pools, disks, and downstream services all queue too.
Ignoring retriesWhen latency rises, clients time out and retry — which raises λ, which raises latency further.

Next flagship

The roofline model & failure at GPU scale

Open →