01 · Why it’s true
A rate limiter protects a service by capping how much work each client can send. The token bucket is the classic design: tokens are added at a steady rate R up to a maximum B; each request takes a token, and requests that find the bucket empty are rejected or delayed.
That allows short bursts of up to B requests while keeping the long-run average at R. It needs only two numbers per client — the token count and the last refill time — so it’s cheap. In a fleet of servers, the hard part is keeping counts shared (for example, in a central store) or splitting the limit between servers.
02 · Picture it
03 · In the interview
- A classic coding and design question: implement a rate limiter.
- Choosing between token bucket, fixed window, and sliding window, and explaining the edge cases.
- Making it work across many servers.
04 · Practice question
“Implement a rate limiter that allows each user 100 requests per minute with short bursts.”
A strong answer covers
- Token bucket per user: capacity 100, refilled at 100 per 60 seconds.
- On each request, add tokens for the time elapsed since the last update (capped at capacity); if at least one token remains, take it and allow, otherwise reject with 429.
- Store only (tokens, last_update) per user, refilling lazily when a request arrives — no background timer needed.
- Across servers: a central store with atomic updates, or split the limit between servers; discuss the latency and accuracy trade-off.