Load Balancing

Overview

A load balancer sits between clients and a pool of servers and decides which server handles each request. Clients talk to one stable address; the balancer spreads the work, detects failed servers, and removes them from rotation.

Load balancing is what makes horizontal scaling possible. Instead of buying one bigger machine, you add more identical machines behind the balancer. It also improves availability because a single server failure no longer takes the service down.

Why you need it

• One server cannot handle peak traffic, and vertical scaling is too expensive or has reached its limit.
• You need zero-downtime deployments: drain a server, update it, and put it back.
• You want the service to survive the loss of a server or an availability zone.
• You need one place to terminate TLS, enforce limits, or route by path or host.

Where it shows up in interviews

Where it is used in real software

Key terms

Backend pool

The set of servers that can serve a request.

Health check

A periodic probe (for example GET /health) that marks servers healthy or unhealthy.

L4 balancing

Routes by IP and port (TCP/UDP) without reading request content. Very fast.

L7 balancing

Reads HTTP data such as path, headers, and cookies to make routing decisions.

Sticky session

Sending the same client to the same server, usually via a cookie.

Connection draining

Letting in-flight requests finish before removing a server.

How a request flows

1. 1
   DNS resolves to the balancer

   The client resolves api.example.com to the balancer's IP, never to an individual server.

2. 2
   Balancer accepts the connection

   It may terminate TLS here so backends receive plain HTTP inside the private network.

3. 3
   Filter to healthy servers

   Servers failing recent health checks are excluded from the candidate list.

4. 4
   Pick a server with the algorithm

   Round robin, least connections, weighted, or hashing chooses one healthy backend.

5. 5
   Forward and relay

   The request goes to the chosen server, and the response is returned to the client. Headers such as X-Forwarded-For preserve the client's IP.

Choosing an algorithm

Three servers. Requests vary from 5 ms reads to 2 s report exports.

With slow report exports mixed in, round robin keeps sending new requests to a server stuck on exports. Least connections steers traffic away from busy servers, which improves tail latency (P99).

Implementation

1upstream api_servers {2    least_conn;                        # algorithm3    server 10.0.1.10:8080 weight=2;    # larger machine4    server 10.0.1.11:8080;5    server 10.0.1.12:8080 max_fails=3 fail_timeout=30s;6    keepalive 64;                      # reuse upstream connections7}8 9server {10    listen 443 ssl;11    server_name api.example.com;12 13    location / {14        proxy_pass http://api_servers;15        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;16        proxy_next_upstream error timeout http_502 http_503;17        proxy_connect_timeout 2s;18        proxy_read_timeout 10s;19    }20}

Complexity and performance

Trade-offs

Variants and related techniques

Common mistakes

• Health checks that only confirm the process is alive.

  Fix: Check critical dependencies carefully, but avoid failing every server at once because a shared database is slow.

• Storing sessions in server memory.

  Fix: Externalize state so any server can handle any request.

• No connection draining during deploys.

  Fix: Stop sending new requests, wait for in-flight ones, then shut down.

• Ignoring slow servers.

  Fix: Use passive health checks (outlier detection) that eject servers with high error rates or latency.

Interview questions

Practice problems