Overview
A distributed lock ensures that only one process across many machines performs a task or modifies a resource at a time, for example running a scheduled job once or preventing two workers from processing the same order. Unlike a local mutex, it must handle process crashes, network delays, and clock differences.
Locks are acquired with a lease (an expiry time) so a crashed holder does not block forever. But a lease alone is not safe: a paused process may resume after its lease expired and another holder took over. Correct designs use fencing tokens, increasing numbers checked by the protected resource, so stale holders are rejected.
Whoever holds the key uses the room. The key automatically becomes invalid after an hour in case someone leaves with it. To stop someone with an expired key from walking in, the room checks a key number and only accepts the newest one.
When to use it
- Ensure a cron job runs on exactly one instance.
- Serialize access to an external resource that lacks its own concurrency control.
- Leader-like roles among workers.
- Prevent duplicate processing when idempotency is not possible.
Where it shows up in interviews
Recognize it when: only one instance may run this job.
- Design a distributed job scheduler
- Design a billing run
Recognize it when: two workers must not modify the same thing.
- Design a ticket booking hold
- Design a collaborative document lock
Where it is used in real software
The common lightweight lock; Redlock extends it to multiple Redis nodes, though its safety is debated.
Provide consensus-backed locks with sessions, sequential nodes, and revision numbers usable as fencing tokens.
A lock service used by Bigtable and GFS for leader election and coarse-grained locks.
Key terms
- Lease
- A lock that expires automatically after a TTL.
- Fencing token
- Monotonically increasing number issued with the lock and checked by the resource.
- Owner token
- A unique value so only the owner can release the lock.
- Lock renewal
- Extending a lease while work continues (heartbeat).
- Redlock
- Acquire a lock on a majority of independent Redis nodes.
How it works, step by step
- 1Acquire atomically with a TTL
SET key owner NX PX 30000, or an etcd lease.
- 2Get a fencing token
An increasing number from the lock service.
- 3Pass the token to the resource
The resource rejects tokens lower than the highest seen.
- 4Renew while working
Extend the lease before it expires.
- 5Release only if owner
Compare-and-delete with the owner token.
STEP 1A acquires the lock with fencing token 33.
Lock implementations
Choose by safety needs
| Implementation | Safety | Performance | Use for |
|---|---|---|---|
| Single Redis SET NX | Lost on failover | Very fast | Efficiency locks (avoid duplicate work) |
| Redlock | Debated under clock skew | Fast | Efficiency locks |
| etcd / ZooKeeper | Strong, with fencing tokens | Slower (consensus) | Correctness locks |
| Database row lock | Strong within one DB | Moderate | Resources in the same database |
NOWImplementation: Single Redis SET NX | Safety: Lost on failover | Performance: Very fast | Use for: Efficiency locks (avoid duplicate work)
Ask whether the lock is for efficiency (occasional duplicate is fine) or correctness (duplicates corrupt data). Correctness locks need consensus and fencing.
Implementation
import { Etcd3 } from "etcd3"; const etcd = new Etcd3({ hosts: process.env.ETCD_HOSTS!.split(",") }); export async function runExclusive(name: string, work: (fencingToken: string) => Promise<void>) { const lock = etcd.lock(`locks/${name}`).ttl(30); // lease renewed automatically while held await lock.acquire(); try { const token = (await lock.leaseId())!; // use the lease ID or key revision as fencing token await work(token); } finally { await lock.release(); }} // Storage side: reject stale tokens// UPDATE jobs SET result = $1, fence = $2 WHERE id = $3 AND fence < $2Complexity and performance
Sub-millisecond.
Milliseconds.
Trade-offs
Consensus-backed locks are safe but slower and need operating etcd or ZooKeeper; Redis locks are fast but can be lost.
Short leases recover quickly after crashes but risk expiring during long work; long leases block others after a crash.
Variants and related techniques
Avoid locks by using version checks on writes.
Make duplicate execution harmless instead of preventing it.
Common mistakes
- Releasing someone else's lock.
Fix: Store a unique owner token and delete only if it matches (atomic script).
- No TTL.
Fix: A crashed holder would block forever.
- Trusting leases for correctness.
Fix: Process pauses can outlive the lease; use fencing tokens.
Interview questions
How would you implement a distributed lock?
Acquire atomically with a TTL and a unique owner ID (Redis SET NX PX for efficiency, etcd or ZooKeeper for correctness), renew the lease while working, release with compare-and-delete, and protect resources with fencing tokens.
What is a fencing token?
A number that increases every time the lock is granted. The protected resource remembers the highest token it has seen and rejects operations with lower tokens, stopping stale lock holders.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| Implement a Redis lock with safe release | Medium | Owner tokens. |
| Design a distributed cron scheduler | Hard | Locks and fencing. |