Overview
A rate limiter controls how many requests a client can make in a given time window. Requests over the limit are rejected, usually with HTTP 429 Too Many Requests, so one client cannot exhaust shared capacity.
Rate limiting protects services from abuse, runaway scripts, brute-force logins, and cost explosions on paid APIs. It is also a fairness tool: every tenant receives its share of capacity.
You receive a token every second, up to a maximum of 10 in your pocket. Each game costs one token. You can play 10 games in a burst, but after that you can only play as fast as tokens arrive.
Why you need it
- Public APIs that must stay available when some clients misbehave.
- Login and OTP endpoints that attackers try to brute-force.
- Expensive operations such as report generation or LLM calls with per-token costs.
- Multi-tenant systems where each plan has a quota.
Where it shows up in interviews
Recognize it when: prevent abuse and overload.
- Design a rate limiter
- Design a public API platform
- Design an API gateway
Recognize it when: per-user or per-tenant limits.
- Design a SaaS with pricing tiers
- Design login brute-force protection
Where it is used in real software
Return rate-limit headers (limit, remaining, reset) and 429 responses when clients exceed quotas.
Uses sliding-window counters at the edge to absorb abusive traffic before it reaches origins.
Provide local token buckets and global rate limiting services backed by Redis.
Key terms
- Limit
- Maximum allowed requests per window, for example 100 per minute.
- Key
- What the limit applies to: user ID, API key, IP address, or endpoint.
- Burst
- Short spike above the average rate that the algorithm allows.
- 429 response
- Standard rejection status, often with a Retry-After header.
Token bucket algorithm
- 1Give each key a bucket
The bucket has a capacity (maximum burst) and a refill rate (sustained requests per second).
- 2Refill lazily
On each request, add (now - lastRefill) x rate tokens, capped at capacity. No background timer is needed.
- 3Spend a token
If at least one token is available, subtract one and allow the request.
- 4Reject otherwise
Return 429 with Retry-After set to the time until the next token.
- 5Share state for many servers
Store buckets in Redis and update them atomically with a Lua script so all servers enforce the same limit.
Comparing algorithms (limit: 10 requests per minute)
A client sends 10 requests at 0:59 and 10 more at 1:01.
| Algorithm | Memory per key | Allows the 20 requests? | Notes |
|---|---|---|---|
| Fixed window counter | 1 counter | Yes, all 20 | Boundary burst: double the limit in 2 seconds |
| Sliding window log | 1 timestamp per request | No, second 10 rejected | Exact but memory-heavy |
| Sliding window counter | 2 counters | Mostly rejected | Weighted estimate, small error |
| Token bucket | Tokens + timestamp | Depends on capacity | Allows controlled bursts; very common |
| Leaky bucket | Queue | Queued and smoothed | Constant output rate; adds delay |
NOWAlgorithm: Fixed window counter | Memory per key: 1 counter | Allows the 20 requests?: Yes, all 20 | Notes: Boundary burst: double the limit in 2 seconds
Fixed windows are simple but allow bursts at window edges. Token bucket is the usual default because it is O(1) in memory and time, and it lets you tune burst size separately from the average rate.
Implementation
class TokenBucket { private tokens: number; private lastRefill = Date.now(); constructor(private capacity: number, private refillPerSecond: number) { this.tokens = capacity; } tryConsume(): boolean { const now = Date.now(); const elapsed = (now - this.lastRefill) / 1000; this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillPerSecond); this.lastRefill = now; if (this.tokens >= 1) { this.tokens -= 1; return true; } return false; }} const buckets = new Map<string, TokenBucket>(); function allow(apiKey: string): boolean { if (!buckets.has(apiKey)) buckets.set(apiKey, new TokenBucket(10, 1)); // burst 10, 1 req/s return buckets.get(apiKey)!.tryConsume();}Complexity and performance
Time and memory per key.
Stores each timestamp inside the window.
One network round trip per request.
Trade-offs
Per-server limits are fast but a client spread across N servers gets N times the limit. Redis gives a global limit at the cost of a network call and a dependency.
If Redis is down, failing open keeps the service usable but unprotected; failing closed protects backends but rejects everyone. Most APIs fail open with a local fallback limit.
Exact sliding logs cost memory. Approximate algorithms are usually good enough.
Variants and related techniques
Different limits by plan, for example free 60/min and paid 1,000/min.
Combine per-IP at the edge, per-user at the gateway, and per-endpoint for expensive operations.
Limit in-flight requests instead of rate, useful for slow operations.
Lower limits automatically when backend latency or error rates rise.
Common mistakes
- Limiting only by IP.
Fix: Many users share an IP behind NAT; prefer API keys or user IDs, and use IP only for anonymous traffic.
- Non-atomic read-modify-write in Redis.
Fix: Use a Lua script or atomic INCR so concurrent requests do not both pass.
- Not telling clients when to retry.
Fix: Return Retry-After and rate-limit headers so well-behaved clients back off.
Interview questions
Which algorithm would you choose and why?
Token bucket: O(1) memory, allows bursts up to capacity, and enforces an average rate. Use sliding window counter if boundary accuracy matters more.
Where should the rate limiter live?
Usually in the API gateway or edge, so rejected requests never reach services. Keep specific business limits, like password attempts, inside the service.
How do you rate limit across many servers?
Keep counters in a shared store such as Redis with atomic scripts, shard by key, and use local token caches if Redis latency is a concern.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| Design a Rate Limiter | Medium | Algorithm choice and distributed state. |
| Logger Rate Limiter | Easy | Per-key timestamps. |
| Design an LLM API token budget | Hard | Limits by tokens, not requests. |