Overview
Throughput is how much work a system completes per unit of time: requests per second, messages per second, or megabytes per second. Latency measures one operation; throughput measures capacity. A system can have low latency but low throughput (one fast worker) or high throughput with high latency (large batches).
Little's Law ties them together: concurrency = throughput x latency. If a service handles 1,000 requests per second and each takes 200 ms, about 200 requests are in flight at any moment. This tells you how many threads, connections, or instances you need.
Latency is how long one car takes to drive from A to B. Throughput is how many cars pass per hour. Adding lanes (parallelism) increases throughput without making any single car faster.
When to use it
- Capacity planning: how many servers for peak traffic?
- Estimating QPS from user numbers in interviews.
- Designing batch jobs and data pipelines.
- Sizing connection pools and worker counts.
Where it shows up in interviews
Recognize it when: estimate load from daily active users.
- Design Twitter: estimate tweets and reads per second
- Design a URL shortener capacity
Recognize it when: ingest millions of events per second.
- Design a metrics pipeline
- Design a log aggregation system
Where it is used in real software
Achieves millions of messages per second per cluster by batching, sequential disk writes, and partitioning.
Tools like k6, Gatling, and Locust measure maximum sustainable throughput before latency degrades.
Retailers plan for Black Friday peaks that can be 5-10x normal traffic.
Key terms
- QPS / RPS
- Queries or requests per second.
- Little's Law
- L = lambda x W: items in system = arrival rate x time in system.
- Peak factor
- Ratio of peak traffic to average (often 2-5x).
- Saturation
- The point where adding load increases latency instead of throughput.
How it works, step by step
- 1Estimate daily volume
DAU x actions per user per day.
- 2Convert to per second
Divide by 86,400 (about 10^5 seconds per day).
- 3Apply the peak factor
Multiply by 2-5x for peak hours.
- 4Divide by per-instance capacity
Measured by load testing, with headroom.
- 5Check concurrency with Little's Law
Size threads, connections, and pools.
Estimating load for a social app
100M daily active users
| Quantity | Calculation | Result |
|---|---|---|
| Feed reads per day | 100M x 20 | 2B |
| Average read QPS | 2B / 86,400 | ~23,000 / s |
| Peak read QPS | x 3 | ~70,000 / s |
| Posts per day | 100M x 0.5 | 50M (~600 / s avg) |
| App servers (2,000 RPS each at 60%) | 70,000 / 1,200 | ~60 instances |
| Concurrency at 50 ms latency | 70,000 x 0.05 | 3,500 in flight |
NOWQuantity: Feed reads per day | Calculation: 100M x 20 | Result: 2B
The read-to-write ratio (about 40:1 here) tells you to invest in caching and read scaling before write scaling.
Implementation
// Capacity helper for interviews and planningfunction capacity({ dau, actionsPerUser, peakFactor, latencyMs, rpsPerInstance, targetUtilization = 0.6 }: { dau: number; actionsPerUser: number; peakFactor: number; latencyMs: number; rpsPerInstance: number; targetUtilization?: number;}) { const avgRps = (dau * actionsPerUser) / 86_400; const peakRps = avgRps * peakFactor; return { avgRps: Math.round(avgRps), peakRps: Math.round(peakRps), instances: Math.ceil(peakRps / (rpsPerInstance * targetUtilization)), inFlight: Math.round(peakRps * (latencyMs / 1000)), // Little's Law };} capacity({ dau: 100e6, actionsPerUser: 20, peakFactor: 3, latencyMs: 50, rpsPerInstance: 2000 });// { avgRps: 23148, peakRps: 69444, instances: 58, inFlight: 3472 }Complexity and performance
Handy for estimates.
Concurrency from rate and latency.
Trade-offs
Larger batches raise throughput (fewer round trips, better compression) but increase latency for each item.
Running at 90% utilization maximizes throughput per dollar but ruins latency and leaves no room for spikes.
Variants and related techniques
Throughput measured in bytes: 1 Gbps is about 125 MB/s.
Useful throughput after retries and errors are excluded.
Common mistakes
- Planning for averages.
Fix: Plan for peak traffic plus headroom.
- Ignoring downstream limits.
Fix: Your throughput is capped by the slowest dependency (database, third-party API).
Interview questions
How do you estimate the QPS of a system?
Daily active users times actions per user, divided by 86,400 seconds, then multiplied by a peak factor of 2-5x. Separate reads and writes because they scale differently.
What is Little's Law and why is it useful?
Concurrency equals throughput times latency. It lets you compute how many concurrent requests, threads, or connections a service needs at a target rate.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| Estimate QPS and storage for a URL shortener | Easy | Envelope math. |
| Size a worker pool for 5,000 jobs/s at 200 ms each | Medium | Little's Law. |