DISTRIBUTED SYSTEMS / SYSTEM CONCEPT BRIEF

Leader / follower

In leader-follower (primary-replica) replication, one node is the leader and accepts all writes.

IntermediatePhase 05 / Topic 6 of 17RequirementsTrade-offsFailure modes
01

Overview

In leader-follower (primary-replica) replication, one node is the leader and accepts all writes. It records changes in a log and streams them to followers, which apply the same changes in the same order. Followers can serve reads, and if the leader fails one of them is promoted.

This model is simple to reason about because there is a single order of writes, so there are no write conflicts. The challenges are the leader as a write bottleneck, replication lag on followers, and safe failover: detecting that the leader is really dead, choosing the most up-to-date follower, and making sure the old leader cannot keep writing.

An orchestra conductor

Only the conductor sets the tempo (writes); musicians (followers) follow it. If the conductor faints, the concertmaster takes over, but you must make sure the original conductor does not wake up and keep conducting a different tempo.

02

When to use it

  • Relational databases and most strongly consistent stores.
  • Read-heavy workloads that benefit from read replicas.
  • Systems that need a single order of writes.
  • Coordination services built on consensus logs.
03

Where it shows up in interviews

Read scaling with replicas

Recognize it when: 100:1 read/write ratio.

  • Design a news site database
  • Scale an e-commerce catalog
Failover design

Recognize it when: the primary database dies.

  • Design a highly available database
  • Design a payment system
04

Where it is used in real software

PostgreSQL and MySQL

Primary with streaming replicas; tools like Patroni or Orchestrator automate failover.

Kafka partitions

Each partition has one leader broker and follower replicas in the in-sync replica (ISR) set.

Redis Sentinel

Monitors a Redis primary and promotes a replica on failure.

05

Key terms

Leader
The only node that accepts writes.
Follower
Replicates the leader's log and may serve reads.
Replication log
Ordered list of changes (WAL, binlog, commit log).
Promotion
Turning a follower into the new leader.
Fencing
Preventing a deposed leader from making changes.
06

How it works, step by step

  1. 1
    Writes go to the leader

    Clients or a proxy route all writes there.

  2. 2
    Leader appends to its log

    Each change gets a position (LSN, offset).

  3. 3
    Followers pull or receive the log

    Apply changes in order.

  4. 4
    Detect leader failure

    Missed heartbeats beyond a timeout.

  5. 5
    Promote and fence

    Choose the most caught-up follower, redirect clients, block the old leader.

07

Failover risks and mitigations

What can go wrong when the leader dies

Step 1 / 4
RiskCauseMitigation
Lost writesAsync follower was behind when promotedSynchronous replica or quorum commit
Split brainOld leader still accepting writesFencing tokens, STONITH, majority leases
False failoverLeader was only slow, not deadReasonable timeouts, multiple observers
Clients writing to old leaderStale DNS or configProxy-based routing, short TTLs

NOWRisk: Lost writes | Cause: Async follower was behind when promoted | Mitigation: Synchronous replica or quorum commit

Failover is the hardest part of leader-follower replication. Automating it safely usually requires a consensus system to decide who the leader is.

08

Implementation

// Simplified follower apply loop with heartbeat-based failure detectionclass Follower {  private appliedOffset = 0;  private lastHeartbeat = Date.now();   onLogEntries(entries: { offset: number; op: Operation }[]) {    for (const entry of entries) {      if (entry.offset !== this.appliedOffset + 1) throw new Error("gap in log: request resync");      applyToStorage(entry.op);      this.appliedOffset = entry.offset;    }    this.lastHeartbeat = Date.now();  }   leaderSuspected(timeoutMs = 3000) {    return Date.now() - this.lastHeartbeat > timeoutMs; // triggers an election, not an instant promotion  }}
09

Complexity and performance

Write throughputLimited by one leader

Shard to scale writes.

Read throughputScales with followers

Subject to lag.

10

Trade-offs

Simplicity vs write scale

One leader means no conflicts but caps writes at one machine.

Sync vs async followers

Sync prevents data loss on failover; async keeps writes fast.

11

Variants and related techniques

Multi-leader

One leader per region with conflict resolution.

Consensus-based (Raft)

Leader elected by majority; commits require majority acknowledgment.

12

Common mistakes

  • Promoting a lagging follower.

    Fix: Pick the follower with the highest applied log position.

  • No fencing of the old leader.

    Fix: Use epochs or fencing tokens so stale leaders are rejected.

13

Interview questions

What happens when the leader fails?

Followers detect missing heartbeats, an election or orchestrator picks the most up-to-date follower, clients are redirected, and the old leader is fenced so it cannot accept writes if it comes back.

How do you scale writes in a leader-follower setup?

Vertically scale the leader, batch writes, and eventually shard so each shard has its own leader.

14

Practice problems

ProblemDifficultyWhat it trains
Draw a failover sequence with timelinesMediumDetection and fencing.
Design read routing with replicasEasyLag-aware reads.