Overview
Resilience is a system's ability to handle failures, overload, and unexpected conditions while continuing to serve users, and to recover quickly. In distributed systems, dependencies will be slow or unavailable, so resilient services are designed to contain failures instead of spreading them.
The standard toolkit combines timeouts (never wait forever), retries with backoff (recover from transient faults), circuit breakers (stop calling a failing dependency), bulkheads (isolate resources), rate limiting and load shedding (protect against overload), fallbacks (degraded but useful responses), and idempotency (make retries safe). Resilience is verified with load tests, chaos experiments, and game days.
Fire doors (bulkheads) contain fires, circuit breakers cut power to faulty wiring, sprinklers (fallbacks) limit damage, and fire drills (chaos tests) make sure everyone knows what to do.
When to use it
- Any service with network dependencies.
- Preventing cascading failures in microservices.
- Handling traffic spikes and partial outages.
- Reliability discussions in interviews.
Where it shows up in interviews
Recognize it when: one slow service takes everything down.
- Design a resilient microservices checkout
- Design an API gateway
Recognize it when: traffic exceeds capacity.
- Design for Black Friday
- Design a public API
Where it is used in real software
Pioneered circuit breakers, bulkheads, and fallbacks at scale; succeeded by Resilience4j.
Provide timeouts, retries, outlier detection, and circuit breaking in the mesh.
Documents Amazon's practices: timeouts, backoff with jitter, load shedding, and static stability.
Key terms
- Cascading failure
- A failure spreading through dependent services.
- Load shedding
- Rejecting excess requests to protect the service.
- Fallback
- Alternative response when a dependency fails.
- Backpressure
- Signaling producers to slow down.
- Graceful degradation
- Reduced functionality instead of failure.
How it works, step by step
- 1Set timeouts on every call
Based on dependency latency percentiles.
- 2Retry carefully
Only idempotent calls, with backoff, jitter, and budgets.
- 3Add circuit breakers
Fail fast when a dependency is unhealthy.
- 4Isolate with bulkheads
Separate pools per dependency.
- 5Define fallbacks and shed load
Cached data, defaults, 503 with Retry-After.
Resilience patterns and what they prevent
A product page calling reviews, recommendations, and inventory
| Pattern | Prevents | Example |
|---|---|---|
| Timeout | Threads stuck waiting | Reviews call capped at 300 ms |
| Retry + backoff | Failing on transient blips | One retry on connection reset |
| Circuit breaker | Hammering a dead service | Recommendations open after 50% errors |
| Bulkhead | One dependency exhausting all threads | Separate pool for inventory |
| Fallback | Blank pages | Show cached reviews or hide the widget |
| Load shedding | Total collapse under overload | Reject low-priority requests at 90% capacity |
NOWPattern: Timeout | Prevents: Threads stuck waiting | Example: Reviews call capped at 300 ms
Patterns work together: timeouts feed circuit breakers, bulkheads contain damage, and fallbacks keep the user experience acceptable.
Implementation
CircuitBreaker breaker = CircuitBreaker.of("recommendations", CircuitBreakerConfig.custom() .failureRateThreshold(50) .slidingWindowSize(20) .waitDurationInOpenState(Duration.ofSeconds(30)) .build()); TimeLimiter timeLimiter = TimeLimiter.of(Duration.ofMillis(300));Bulkhead bulkhead = Bulkhead.of("recommendations", BulkheadConfig.custom().maxConcurrentCalls(20).build());Retry retry = Retry.of("recommendations", RetryConfig.custom() .maxAttempts(2) .intervalFunction(IntervalFunction.ofExponentialRandomBackoff(100, 2.0)) .build()); Supplier<List<Product>> call = () -> recommendationsClient.similar(productId);Supplier<List<Product>> resilient = Decorators.ofSupplier(call) .withBulkhead(bulkhead) .withCircuitBreaker(breaker) .withRetry(retry) .withFallback(List.of(Exception.class), e -> List.of()) // hide the widget .decorate();Complexity and performance
Not arbitrary.
Retry at one layer.
Trade-offs
Each pattern adds configuration and behavior to test; misconfigured retries or breakers can cause outages themselves.
Fallbacks to cached or default data keep pages up but show stale information.
Variants and related techniques
Keep operating with last-known-good data when control planes fail.
Isolate customers into independent cells.
Common mistakes
- No timeouts (library defaults can be infinite).
Fix: Set connect and request timeouts explicitly.
- Retries without circuit breakers.
Fix: Retries amplify load on failing services.
- Untested fallbacks.
Fix: Exercise them with chaos testing.
Interview questions
How do you prevent cascading failures?
Timeouts on all calls, limited retries with backoff and jitter, circuit breakers to fail fast, bulkheads to isolate resources, load shedding and rate limits for overload, and fallbacks for non-critical features.
How would you make a product page resilient to a slow recommendations service?
Call it with a short timeout inside its own bulkhead, wrap it in a circuit breaker, fall back to cached or no recommendations, and render the core product data regardless.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| Add timeouts, retries, and a breaker to a client | Medium | Composition order. |
| Design resilience for a checkout flow | Hard | Critical vs optional dependencies. |