Overview
Leader election chooses exactly one node among many to take a special role: accept writes, run scheduled jobs, assign work, or coordinate the cluster. When the leader fails, a new leader must be elected quickly, and the system must guarantee that two nodes never act as leader at the same time.
Robust leader election is almost always built on a consensus system (Raft, ZooKeeper, etcd) or a strongly consistent lease (Kubernetes Lease objects, DynamoDB conditional writes). The elected leader holds a lease it must renew, and every leader action carries an epoch or fencing token so actions from a deposed leader are rejected.
The team votes, and whoever gets a majority wears the armband for this season (term). If the captain is injured, a new vote happens. The armband has a season number, so an old captain cannot claim authority after being replaced.
When to use it
- Single-writer databases and partition leaders.
- Controllers that must run once, like Kubernetes controllers or schedulers.
- Assigning shards or work to workers.
- Cluster coordination and membership management.
Where it shows up in interviews
Recognize it when: only one instance may run the scheduler.
- Design a distributed cron
- Design a Kubernetes controller
Recognize it when: pick a new database primary after failure.
- Design automatic database failover
- Design Kafka partition leadership
Where it is used in real software
Controllers like kube-scheduler use Lease resources in etcd to elect one active instance.
Kafka controllers use Raft to elect an active controller, which then assigns partition leaders.
Uses etcd, Consul, or ZooKeeper to elect the PostgreSQL primary and manage failover.
Key terms
- Lease
- Leadership that expires unless renewed.
- Epoch / term
- Increasing number identifying each leadership period.
- Heartbeat
- Periodic signal proving the leader is alive.
- Bully algorithm
- Classic algorithm where the highest ID wins; not partition safe.
- Fencing
- Rejecting actions tagged with an old epoch.
How it works, step by step
- 1Candidates try to acquire leadership
Conditional write or consensus vote.
- 2Winner receives an epoch
Higher than any previous leader's.
- 3Leader renews its lease
Heartbeats well before expiry.
- 4Followers watch
On lease expiry, they compete again.
- 5Fence stale leaders
Downstream systems reject lower epochs; leaders step down if renewal fails.
Leader election approaches
Choose by safety requirements
| Approach | Mechanism | Safe under partitions? |
|---|---|---|
| Bully / ring algorithms | Highest ID wins | No |
| Redis SET NX lease | Key with TTL | Weak (failover can lose the key) |
| DynamoDB conditional write | Lease row with version | Yes, with fencing |
| etcd / ZooKeeper / Raft | Consensus | Yes |
NOWApproach: Bully / ring algorithms | Mechanism: Highest ID wins | Safe under partitions?: No
Use a consensus-backed primitive when two leaders would corrupt data; lighter options are fine when duplicates are just wasted work.
Implementation
lock := &resourcelock.LeaseLock{ LeaseMeta: metav1.ObjectMeta{Name: "report-scheduler", Namespace: "jobs"}, Client: clientset.CoordinationV1(), LockConfig: resourcelock.ResourceLockConfig{Identity: podName},} leaderelection.RunOrDie(ctx, leaderelection.LeaderElectionConfig{ Lock: lock, LeaseDuration: 15 * time.Second, // followers wait this long before taking over RenewDeadline: 10 * time.Second, // leader must renew within this window RetryPeriod: 2 * time.Second, ReleaseOnCancel: true, Callbacks: leaderelection.LeaderCallbacks{ OnStartedLeading: func(ctx context.Context) { runScheduler(ctx) }, OnStoppedLeading: func() { log.Println("lost leadership, stopping work") }, },})Complexity and performance
Seconds typically.
Cheap.
Trade-offs
Short leases recover quickly but may elect a new leader while the old one is merely slow.
Using etcd or ZooKeeper adds infrastructure but gives correctness.
Variants and related techniques
Avoid election entirely with quorums or partitioned ownership.
Different leaders per partition spread the load.
Common mistakes
- Leader keeps working after losing the lease.
Fix: Stop work when renewal fails and check the epoch before each action.
- Clock-based lease assumptions.
Fix: Account for clock drift; use monotonic clocks and safety margins.
Interview questions
How do you ensure only one instance of a scheduled job runs?
Elect a leader using a consensus-backed lease (Kubernetes Lease, etcd, ZooKeeper). Only the leader runs jobs, renews the lease regularly, stops if renewal fails, and tags work with an epoch so a stale leader's writes are rejected.
What happens if the leader is partitioned from the others?
It cannot renew its lease with the majority, so it must step down when the lease expires; the majority side elects a new leader with a higher epoch, and fencing rejects any late actions from the old leader.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| Implement leader election with a conditional write | Medium | Leases. |
| Design partition leadership for a message broker | Hard | Controller and ISR. |