Overview
Fault tolerance is a system's ability to keep working correctly when some of its components fail. Faults are normal at scale: disks die, processes crash, networks drop packets, deployments go wrong, and dependencies slow down. A fault-tolerant design expects these faults and contains them so they do not become user-visible failures.
The main techniques are redundancy (replicas, multiple zones), isolation (bulkheads, separate failure domains), detection (health checks, timeouts), recovery (retries, failover, restarts), and degradation (fallbacks, cached responses). Chaos engineering tests these mechanisms by injecting failures deliberately.
Planes have multiple engines, hydraulic systems, and flight computers. A single failure is expected and handled by design, and pilots train for failures in simulators just like chaos experiments.
When to use it
- Every production system beyond a prototype.
- Designing for zone or region outages.
- Protecting core flows from non-critical dependencies.
- Reliability sections of system design interviews.
Where it shows up in interviews
Recognize it when: what if this component fails?
- Design a notification system
- Design a payment service
Recognize it when: one failure must not take everything down.
- Design a multi-tenant platform
- Design microservices for e-commerce
Where it is used in real software
Randomly terminates production instances to ensure services survive failures; Chaos Kong simulates region failures.
Services are split into independent cells so a failure affects only a fraction of customers.
Restarts crashed containers, replaces failed pods, and reschedules workloads from failed nodes.
Key terms
- Fault vs failure
- A component deviating vs the system failing to serve users.
- Failure domain
- A scope that fails together: host, rack, zone, region.
- Blast radius
- How much of the system a single failure affects.
- Graceful degradation
- Reduced functionality instead of total failure.
- Chaos engineering
- Deliberately injecting faults to verify resilience.
How it works, step by step
- 1Enumerate failure modes
Crash, slow, wrong answers, partition, overload, bad deploy.
- 2Add redundancy across failure domains
Replicas in multiple zones.
- 3Detect quickly
Timeouts, health checks, alerts on SLOs.
- 4Contain
Circuit breakers, bulkheads, cells, rate limits.
- 5Recover and learn
Automated failover, rollbacks, postmortems, chaos tests.
Failure modes and responses
An e-commerce product page
| Failure | Detection | Response |
|---|---|---|
| App instance crash | Health check fails | LB removes it; orchestrator restarts |
| Recommendations service slow | Timeout at 200 ms | Circuit opens; hide the widget |
| Primary DB fails | Replica loses heartbeat | Promote standby; reads continue from cache |
| Bad deployment | Error rate SLO alert | Automatic rollback of canary |
| Availability zone outage | Many checks fail | Traffic shifts to other zones |
NOWFailure: App instance crash | Detection: Health check fails | Response: LB removes it; orchestrator restarts
Each failure is expected, detected, and handled automatically; the product page stays up, possibly with fewer features.
Implementation
// Graceful degradation: the page works even if optional dependencies failexport async function productPage(id: string) { const product = await getProduct(id); // critical: failure is an error const [reviews, recs] = await Promise.allSettled([ withTimeout(reviewsClient.top(id), 300), withTimeout(recsClient.similar(id), 200), ]); return { product, reviews: reviews.status === "fulfilled" ? reviews.value : await cache.get(`reviews:${id}`) ?? [], recommendations: recs.status === "fulfilled" ? recs.value : [], // hide widget on failure };}Complexity and performance
Majority consensus needs 2f + 1.
Malicious or arbitrary failures.
Trade-offs
Redundancy and failover automation cost money and add moving parts that can fail too.
Retries recover from transient faults but can amplify overload; combine with backoff and circuit breakers.
Variants and related techniques
Independent copies of the stack, each serving a subset of users.
Systems keep working with their last known state when control planes fail.
Common mistakes
- Correlated failures.
Fix: Replicas in the same rack or zone, or deploying a bad config everywhere at once, defeat redundancy.
- Untested failover.
Fix: Run game days and chaos experiments.
- Retry storms.
Fix: Exponential backoff with jitter and retry budgets.
Interview questions
How do you make a system fault tolerant?
Remove single points of failure with redundancy across zones, detect failures with timeouts and health checks, contain them with circuit breakers and bulkheads, recover automatically with failover and restarts, and degrade non-critical features gracefully.
What is blast radius and how do you reduce it?
The portion of users or functionality affected by one failure. Reduce it with cells or shards, staged deployments, bulkheads, and isolating tenants and dependencies.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| List failure modes for a URL shortener | Easy | Enumeration. |
| Design a fault-tolerant notification system | Medium | Retries and DLQs. |
| Design a cell-based architecture for a SaaS | Hard | Blast radius. |