Overview
Split brain happens when a cluster is divided by a partition or a false failure detection and two parts both believe they are in charge, for example two database nodes both acting as primary and accepting writes. The data diverges, and merging it later can be impossible without data loss.
Prevention relies on requiring a majority (quorum) before acting as leader, fencing the old leader so it cannot write (fencing tokens, revoking storage access, or STONITH, 'shoot the other node in the head'), and using an odd number of voters or a tie-breaking witness.
If the radio fails and each half of the crew picks its own captain, the ship receives contradictory orders. The fix is a rule that only the group with most of the crew can pick the captain, and the old captain's orders are ignored once replaced.
When to use it
- Designing failover for databases and clustered services.
- Evaluating two-node clusters and multi-region deployments.
- Explaining why majority quorums matter.
- Postmortems of data divergence incidents.
Where it shows up in interviews
Recognize it when: what if the old primary comes back?
- Design database high availability
- Design a leader-based queue
Recognize it when: two regions accept writes.
- Design a global inventory system
- Design multi-region failover
Where it is used in real software
A 43-second network partition led to writes on both coasts' MySQL clusters, and reconciliation took about 24 hours of degraded service.
Linux HA clusters power off the other node via a fencing device before taking over its resources.
SQL Server, Azure, and MongoDB arbiters provide a tie-breaking vote for two-node setups.
Key terms
- Split brain
- Multiple nodes acting as leader simultaneously.
- Fencing
- Blocking the old leader from resources.
- STONITH
- Forcibly powering off the other node.
- Witness / arbiter
- A lightweight voter to break ties.
- Epoch
- Leadership generation number used to reject stale leaders.
How it works, step by step
- 1Require a majority to lead
The minority side cannot elect or keep a leader.
- 2Use odd voter counts
3 or 5 nodes, or 2 nodes plus a witness.
- 3Fence before promotion
Revoke the old leader's access or power it off.
- 4Tag writes with epochs
Storage rejects writes from older epochs.
- 5Detect divergence
Compare logs after healing and alert.
STEP 1Normal: A is primary, B replicates from A.
Split-brain defenses
Layers of protection
| Defense | Prevents | Cost |
|---|---|---|
| Majority quorum | Minority electing a leader | Need 3+ voters |
| Witness node | Ties in 2-node clusters | Small extra node |
| Fencing tokens / epochs | Stale leader writes | Storage must check tokens |
| STONITH | Old node doing anything | Fencing hardware or cloud API |
NOWDefense: Majority quorum | Prevents: Minority electing a leader | Cost: Need 3+ voters
Majority quorums prevent electing two leaders; fencing handles the old leader that has not realized it lost leadership.
Implementation
-- Epoch-based fencing at the storage layerCREATE TABLE cluster_epoch (id INT PRIMARY KEY CHECK (id = 1), epoch BIGINT NOT NULL); -- New leader bumps the epoch during promotionUPDATE cluster_epoch SET epoch = epoch + 1 WHERE id = 1 RETURNING epoch; -- e.g. 8 -- Every leader write includes its epoch; stale leaders (epoch 7) affect 0 rowsUPDATE accountsSET balance = balance - 100WHERE id = 42 AND (SELECT epoch FROM cluster_epoch WHERE id = 1) = 7;Complexity and performance
Majority always unique.
Without a witness.
Trade-offs
Refusing to promote without a majority prevents split brain but can mean downtime when the majority is unreachable.
Automation recovers fast but risks split brain on false detection; manual failover is safe but slow.
Variants and related techniques
SCSI reservations or cloud volume detach prevent the old node from writing.
The leader stops serving if it cannot renew within the lease.
Common mistakes
- Two-node clusters with automatic failover.
Fix: Add a witness or use three nodes.
- Relying only on timeouts.
Fix: A slow leader is not a dead leader; fence it.
Interview questions
What is split brain and how do you prevent it?
Two nodes both acting as leader after a partition, causing divergent writes. Prevent it with majority quorums for election, an odd number of voters, fencing tokens checked by storage, and fencing or powering off the old leader before promotion.
Why are two-node clusters dangerous?
When they lose contact, neither side has a majority, so either both stop (no availability) or both continue (split brain). A third voter breaks the tie.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| Analyze the GitHub 2018 incident | Medium | Failure timeline. |
| Design safe failover for a two-region database | Hard | Witness and fencing. |