DISTRIBUTED SYSTEMS / SYSTEM CONCEPT BRIEF

Split brain

Split brain happens when a cluster is divided by a partition or a false failure detection and two parts both believe they are in charge, for example two database nodes both acting as primary and accepting writes.

AdvancedPhase 05 / Topic 16 of 17RequirementsTrade-offsFailure modes
01

Overview

Split brain happens when a cluster is divided by a partition or a false failure detection and two parts both believe they are in charge, for example two database nodes both acting as primary and accepting writes. The data diverges, and merging it later can be impossible without data loss.

Prevention relies on requiring a majority (quorum) before acting as leader, fencing the old leader so it cannot write (fencing tokens, revoking storage access, or STONITH, 'shoot the other node in the head'), and using an odd number of voters or a tie-breaking witness.

Two captains on one ship

If the radio fails and each half of the crew picks its own captain, the ship receives contradictory orders. The fix is a rule that only the group with most of the crew can pick the captain, and the old captain's orders are ignored once replaced.

02

When to use it

  • Designing failover for databases and clustered services.
  • Evaluating two-node clusters and multi-region deployments.
  • Explaining why majority quorums matter.
  • Postmortems of data divergence incidents.
03

Where it shows up in interviews

Failover safety

Recognize it when: what if the old primary comes back?

  • Design database high availability
  • Design a leader-based queue
Multi-region active-active

Recognize it when: two regions accept writes.

  • Design a global inventory system
  • Design multi-region failover
04

Where it is used in real software

GitHub 2018 incident

A 43-second network partition led to writes on both coasts' MySQL clusters, and reconciliation took about 24 hours of degraded service.

Pacemaker STONITH

Linux HA clusters power off the other node via a fencing device before taking over its resources.

Witness nodes

SQL Server, Azure, and MongoDB arbiters provide a tie-breaking vote for two-node setups.

05

Key terms

Split brain
Multiple nodes acting as leader simultaneously.
Fencing
Blocking the old leader from resources.
STONITH
Forcibly powering off the other node.
Witness / arbiter
A lightweight voter to break ties.
Epoch
Leadership generation number used to reject stale leaders.
06

How it works, step by step

  1. 1
    Require a majority to lead

    The minority side cannot elect or keep a leader.

  2. 2
    Use odd voter counts

    3 or 5 nodes, or 2 nodes plus a witness.

  3. 3
    Fence before promotion

    Revoke the old leader's access or power it off.

  4. 4
    Tag writes with epochs

    Storage rejects writes from older epochs.

  5. 5
    Detect divergence

    Compare logs after healing and alert.

Split brain in a two-node cluster
Step 1 / 4
Clients
Node A (primary)
Node B (standby)

STEP 1Normal: A is primary, B replicates from A.

07

Split-brain defenses

Layers of protection

Step 1 / 4
DefensePreventsCost
Majority quorumMinority electing a leaderNeed 3+ voters
Witness nodeTies in 2-node clustersSmall extra node
Fencing tokens / epochsStale leader writesStorage must check tokens
STONITHOld node doing anythingFencing hardware or cloud API

NOWDefense: Majority quorum | Prevents: Minority electing a leader | Cost: Need 3+ voters

Majority quorums prevent electing two leaders; fencing handles the old leader that has not realized it lost leadership.

08

Implementation

-- Epoch-based fencing at the storage layerCREATE TABLE cluster_epoch (id INT PRIMARY KEY CHECK (id = 1), epoch BIGINT NOT NULL); -- New leader bumps the epoch during promotionUPDATE cluster_epoch SET epoch = epoch + 1 WHERE id = 1 RETURNING epoch;  -- e.g. 8 -- Every leader write includes its epoch; stale leaders (epoch 7) affect 0 rowsUPDATE accountsSET balance = balance - 100WHERE id = 42  AND (SELECT epoch FROM cluster_epoch WHERE id = 1) = 7;
09

Complexity and performance

Voters for f failures2f + 1

Majority always unique.

Two-node clusterCannot tolerate partitions safely

Without a witness.

10

Trade-offs

Availability vs safety

Refusing to promote without a majority prevents split brain but can mean downtime when the majority is unreachable.

Automatic vs manual failover

Automation recovers fast but risks split brain on false detection; manual failover is safe but slow.

11

Variants and related techniques

Disk-based fencing

SCSI reservations or cloud volume detach prevent the old node from writing.

Lease-based leadership

The leader stops serving if it cannot renew within the lease.

12

Common mistakes

  • Two-node clusters with automatic failover.

    Fix: Add a witness or use three nodes.

  • Relying only on timeouts.

    Fix: A slow leader is not a dead leader; fence it.

13

Interview questions

What is split brain and how do you prevent it?

Two nodes both acting as leader after a partition, causing divergent writes. Prevent it with majority quorums for election, an odd number of voters, fencing tokens checked by storage, and fencing or powering off the old leader before promotion.

Why are two-node clusters dangerous?

When they lose contact, neither side has a majority, so either both stop (no availability) or both continue (split brain). A third voter breaks the tie.

14

Practice problems

ProblemDifficultyWhat it trains
Analyze the GitHub 2018 incidentMediumFailure timeline.
Design safe failover for a two-region databaseHardWitness and fencing.