Overview
Percentiles describe the distribution of latency. P50 (the median) means half of requests are faster; P95 means 95% are faster and 5% are slower; P99 means 1 in 100 requests is slower. Averages hide slow outliers, so latency goals (SLOs) are written in percentiles such as 'P99 below 300 ms'.
Tail latency matters more than it seems. A page that makes 20 backend calls will see at least one P99-slow call in about 18% of page loads. At scale, the slowest 1% affects many users, and the causes (garbage collection pauses, cold caches, overloaded nodes, retries) are different from what makes the median slow.
Most pizzas arrive in 25 minutes (P50), almost all within 40 (P95), but one in a hundred takes 90 minutes (P99). Customers remember the 90-minute pizza, and if you order for a party from five shops, the chance one is very late goes up.
When to use it
- Defining SLOs and alerts.
- Comparing performance before and after a change.
- Understanding fan-out systems where tails compound.
- Explaining why averages are misleading.
Where it shows up in interviews
Recognize it when: what does 'fast enough' mean for this API?
- Define SLOs for a checkout service
- Design monitoring for an API platform
Recognize it when: one request calls many services or shards.
- Design search across 100 shards
- Design a microservice homepage
Where it is used in real software
A Google paper by Dean and Barroso showed how tail latency dominates large fan-out systems and introduced hedged requests.
Prometheus histograms, HdrHistogram, and Datadog distributions compute percentiles across servers correctly.
Google SRE defines SLIs and SLOs in percentiles and uses error budgets for release decisions.
Key terms
- Percentile
- The value below which a given percentage of observations fall.
- Tail latency
- High percentiles such as P99 and P99.9.
- SLO
- Service level objective, for example 99% of requests under 300 ms.
- Histogram
- Bucketed counts used to compute percentiles efficiently.
How it works, step by step
- 1Record every request's latency
In a histogram, not just an average.
- 2Compute percentiles over windows
For example per minute and per day.
- 3Aggregate histograms, not percentiles
Averaging P99s across servers is mathematically wrong.
- 4Set SLOs per endpoint
Critical paths get stricter targets.
- 5Investigate the tail separately
GC pauses, noisy neighbors, cold caches, lock contention, retries.
Why averages lie
10 requests: nine at 50 ms, one at 2,000 ms
| Metric | Value | What it says |
|---|---|---|
| Average | 245 ms | Looks fine, describes no real request |
| P50 | 50 ms | Typical experience |
| P90 | 50 ms | Still fine |
| P99 / max | 2,000 ms | Some users wait 2 seconds |
NOWMetric: Average | Value: 245 ms | What it says: Looks fine, describes no real request
Fan-out makes it worse: with 20 calls per page, P(at least one call slower than P99) = 1 - 0.99^20, about 18%.
Implementation
# P99 latency per endpoint over 5 minutes, aggregated correctly across all podshistogram_quantile( 0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))) # SLO: fraction of requests faster than 300 mssum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))/sum(rate(http_request_duration_seconds_count[5m]))Complexity and performance
n calls each fast with probability p.
Not O(requests).
Trade-offs
Hedged requests and over-provisioning reduce P99 but increase load and cost.
P99.9 catches rare issues but needs lots of data to be stable; P95 is noisier-proof but hides rare stalls.
Variants and related techniques
Send a backup request after the P95 time and use whichever answers first.
A single score based on satisfied, tolerating, and frustrated thresholds.
Common mistakes
- Averaging percentiles across servers.
Fix: Aggregate raw histograms, then compute percentiles.
- Only tracking averages.
Fix: Alert on P95 / P99.
Interview questions
Why does P99 matter for systems with fan-out?
Each user request touches many backends; the chance that at least one is in its slow tail grows quickly with the number of calls, so backend P99 becomes the user's typical experience.
How do you reduce tail latency?
Remove causes (GC tuning, avoid lock contention, keep caches warm), keep headroom, use timeouts with hedged or backup requests, and isolate slow workloads.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| Compute P50/P95/P99 from a latency list | Easy | Definition. |
| Design SLOs and alerts for a checkout flow | Medium | Percentile targets. |