Latency - System Design Guide

Overview

Latency is the time a single operation takes, from request to response. It is made of network time (propagation, round trips), queueing time (waiting for a busy resource), and service time (actual work). Users experience latency directly, so it is usually the most visible performance metric.

Knowing rough numbers for common operations lets you estimate designs quickly: memory access is nanoseconds, a same-datacenter round trip is about half a millisecond, a disk seek is milliseconds, and a cross-continent round trip is about 100 ms. Designs reduce latency by avoiding round trips, moving data closer (caches, CDNs), and avoiding queueing (headroom, parallelism).

Commute time

Your commute is driving time (service), distance (propagation), and traffic jams (queueing). A faster car helps a little; living closer or avoiding rush hour helps much more.

When to use it

Back-of-the-envelope estimates in interviews.
Setting SLOs for user-facing endpoints.
Deciding where to cache and where to place servers.
Diagnosing slow requests.

Where it shows up in interviews

Latency budget

Recognize it when: an endpoint must respond within 200 ms.

Design typeahead search
Design a payment authorization flow

Global latency

Recognize it when: users worldwide.

Design a global chat app
Design a multiplayer game

Where it is used in real software

Latency numbers every programmer should know

A widely shared table (popularized by Jeff Dean) compares cache, memory, SSD, network, and disk latencies across orders of magnitude.

Typeahead

Search suggestions need to appear within about 100 ms of each keystroke, which forces in-memory indexes and edge serving.

Trading systems

High-frequency traders pay for microwave links and co-location to shave microseconds.

Key terms

Propagation delay: Time for a signal to travel the distance.
Queueing delay: Time waiting for a busy resource; explodes near 100% utilization.
Service time: Time spent doing the work.
RTT: Round-trip time between two points.
Tail latency: Latency at high percentiles (P99, P999).

How it works, step by step

1
Break the request into hops
Client to edge, edge to service, service to cache or DB, and back.
2
Assign rough numbers
Use standard latency figures per hop.
3
Count round trips
Sequential calls add; parallel calls take the maximum.
4
Remove the largest contributors
Cache, colocate, parallelize, or eliminate round trips.
5
Keep utilization moderate
Queueing grows sharply above about 70-80% busy.

Approximate latency numbers

Orders of magnitude, not exact values

Operation	Latency	Relative to 1 ns
L1 cache reference	~1 ns	1x
Main memory reference	~100 ns	100x
Read 1 MB sequentially from memory	~10 us	10,000x
SSD random read	~100 us	100,000x
Round trip within a data center	~0.5 ms	500,000x
Disk seek (HDD)	~5-10 ms	10,000,000x
Round trip US to Europe	~70-100 ms	100,000,000x

NOWOperation: L1 cache reference | Latency: ~1 ns | Relative to 1 ns: 1x

Network round trips dwarf computation. Serving from memory in the same data center is thousands of times faster than crossing an ocean.

Implementation

1// Measure latency of an operation and report percentiles2async function measure<T>(label: string, fn: () => Promise<T>, runs = 200) {3  const samples: number[] = [];4  for (let i = 0; i < runs; i++) {5    const start = performance.now();6    await fn();7    samples.push(performance.now() - start);8  }9  samples.sort((a, b) => a - b);10  const pct = (p: number) => samples[Math.min(samples.length - 1, Math.floor((p / 100) * samples.length))];11  console.log(label, { p50: pct(50).toFixed(1), p95: pct(95).toFixed(1), p99: pct(99).toFixed(1) });12}13 14await measure("redis get", () => redis.get("product:42"));15await measure("postgres query", () => db.query("SELECT * FROM products WHERE id = $1", [42]));

Complexity and performance

Parallel callsmax(latencies)

Latency of the slowest.

Sequential callssum(latencies)

Avoid chains.

Queueing near saturationgrows toward infinity

As utilization approaches 100%.

Trade-offs

Latency vs throughput

Batching improves throughput but adds waiting time to individual requests.

Latency vs consistency

Reading from a nearby replica is faster but may be stale; synchronous replication improves consistency but adds latency.

Variants and related techniques

Hedged requests

Send a duplicate request if the first is slow and take the fastest answer, to cut tail latency.

Edge computing

Run logic near users to cut round trips.

Common mistakes

Ignoring round trips in estimates.
Fix: Count every network hop; it usually dominates.
Running resources near 100% utilization.
Fix: Keep headroom; queueing makes latency explode.

Interview questions

What contributes to request latency?

Network propagation and round trips, queueing when resources are busy, and processing time in each service, cache, and database along the path.

How do you reduce latency for global users?

Serve from nearby locations (CDN, regional deployments), cache aggressively, reduce round trips (connection reuse, batching), and replicate read data close to users.

Practice problems

Problem	Difficulty	What it trains
Estimate latency of a request touching cache, DB, and 2 services	Easy	Budgeting.
Design typeahead with a 100 ms budget	Medium	Where each ms goes.