Resilience Engineering
Timeouts, retries with backoff and jitter, circuit breakers, bulkheads, load shedding and graceful degradation, plus how to verify resilience with chaos experiments.
What it is
Resilience is the ability to absorb, adapt to and recover from failures and unexpected load while continuing to provide acceptable service. Fault tolerance handles known component failures; resilience also covers overload, cascading failures and partial degradation.
Core mechanisms
| Mechanism | Purpose | Key setting |
|---|---|---|
| Timeout | Stop waiting on a slow dependency | Set below the caller's own timeout; budget across the chain |
| Retry with exponential backoff + jitter | Recover from transient faults | Max attempts 2–3, only for idempotent ops and retryable errors |
| Circuit breaker | Stop calling a failing dependency, fail fast | Failure-rate threshold, open duration, half-open probes |
| Bulkhead | Isolate resources so one dependency cannot exhaust all threads/connections | Separate pools per dependency |
| Rate limiting / load shedding | Reject excess work early to protect the core | Shed lowest-priority traffic first |
| Fallback / graceful degradation | Return a reduced but useful answer | Cached data, defaults, feature off |
| Backpressure | Slow producers when consumers lag | Bounded queues, reactive streams |
flowchart LR
Req[Incoming request] --> RL{Rate limit /<br/>priority}
RL -- shed --> R429[429 / degraded response]
RL -- admit --> TO[Timeout budget]
TO --> CB{Circuit<br/>breaker}
CB -- open --> FB[Fallback:<br/>cached / default]
CB -- closed --> BH[Bulkhead pool<br/>per dependency]
BH --> RT[Retry with<br/>backoff + jitter]
RT --> DEP[(Dependency)]Retry storms and why jitter matters
When a dependency recovers, thousands of clients retrying on the same schedule create synchronized spikes that knock it over again. Exponential backoff spreads attempts out; jitter (randomizing the delay) desynchronizes clients. Also cap retries at every layer: three layers each retrying three times means 27 calls to the bottom service.
Real-world example
A product page calls pricing, inventory, reviews and recommendations. Pricing and inventory are essential; reviews and recommendations are not. With 300 ms total budget, each optional call gets a 100 ms timeout and its own bulkhead. If recommendations fail, the circuit opens and the page renders without that carousel. Under extreme load, the edge sheds anonymous bot traffic first, then disables recommendations, protecting checkout.
Advantages and disadvantages
| Advantages | Disadvantages |
|---|---|
| Prevents cascading failures | Misconfigured retries amplify outages |
| Keeps core journeys working | Fallbacks can hide real problems if not monitored |
| Predictable behaviour under overload | More configuration to tune and test |
Technology options
- Libraries: Resilience4j (Java), Spring Cloud Circuit Breaker, Polly (.NET), failsafe-go.
- Infrastructure: Envoy/Istio (retries, timeouts, outlier detection), API gateways (rate limits), AWS App Mesh successor options such as VPC Lattice.
- Testing: AWS Fault Injection Service, Azure Chaos Studio, Chaos Mesh, LitmusChaos, Gremlin, Toxiproxy.
Java example: Resilience4j with Spring Boot
@Service
public class PricingClient {
private final RestClient rest;
public PricingClient(RestClient.Builder builder) {
this.rest = builder.baseUrl("http://pricing").build();
}
@CircuitBreaker(name = "pricing", fallbackMethod = "cachedPrice")
@Retry(name = "pricing")
@TimeLimiter(name = "pricing")
public CompletableFuture<Price> price(String sku) {
return CompletableFuture.supplyAsync(() ->
rest.get().uri("/prices/{sku}", sku).retrieve().body(Price.class));
}
private CompletableFuture<Price> cachedPrice(String sku, Throwable t) {
return CompletableFuture.completedFuture(PriceCache.lastKnown(sku));
}
}resilience4j:
circuitbreaker:
instances:
pricing:
slidingWindowSize: 50
failureRateThreshold: 50
waitDurationInOpenState: 10s
permittedNumberOfCallsInHalfOpenState: 5
retry:
instances:
pricing:
maxAttempts: 3
waitDuration: 100ms
enableExponentialBackoff: true
exponentialBackoffMultiplier: 2
enableRandomizedWait: true
timelimiter:
instances:
pricing:
timeoutDuration: 150ms[!VERSION] Annotation names and property keys follow Resilience4j 2.x with Spring Boot 3. Check the release notes when upgrading to Spring Boot 4.
Architect-level considerations
[!ARCHITECT] Define a timeout budget for each user journey and allocate it across hops. Decide which dependencies are critical vs optional and specify the degraded behaviour for each in the design.
[!EA] Standardize resilience defaults (timeouts, retry policy, circuit breaker settings) in a shared library or mesh configuration so every team does not reinvent them.
Interview questions and answers
Where should retries live: client, gateway, or mesh?
In one place per hop, ideally closest to the caller that knows whether the operation is idempotent. Avoid retries at multiple layers for the same call. Meshes are good for transport-level retries on idempotent methods; business-level retries belong in the application.
Explain the circuit breaker states.
Closed: calls flow, failures are counted. Open: calls fail fast for a cool-down period. Half-open: a few trial calls are allowed; success closes the breaker, failure reopens it.
What is load shedding and how is it different from rate limiting?
Rate limiting enforces a per-client quota regardless of system health. Load shedding drops work based on the system's current capacity and request priority to keep it healthy, for example rejecting batch traffic when CPU or queue depth crosses a threshold.