Resilience Engineering

Timeouts, retries with backoff and jitter, circuit breakers, bulkheads, load shedding and graceful degradation, plus how to verify resilience with chaos experiments.

What it is

Resilience is the ability to absorb, adapt to and recover from failures and unexpected load while continuing to provide acceptable service. Fault tolerance handles known component failures; resilience also covers overload, cascading failures and partial degradation.

Core mechanisms

Mechanism Purpose Key setting
Timeout Stop waiting on a slow dependency Set below the caller's own timeout; budget across the chain
Retry with exponential backoff + jitter Recover from transient faults Max attempts 2–3, only for idempotent ops and retryable errors
Circuit breaker Stop calling a failing dependency, fail fast Failure-rate threshold, open duration, half-open probes
Bulkhead Isolate resources so one dependency cannot exhaust all threads/connections Separate pools per dependency
Rate limiting / load shedding Reject excess work early to protect the core Shed lowest-priority traffic first
Fallback / graceful degradation Return a reduced but useful answer Cached data, defaults, feature off
Backpressure Slow producers when consumers lag Bounded queues, reactive streams
flowchart LR
  Req[Incoming request] --> RL{Rate limit /<br/>priority}
  RL -- shed --> R429[429 / degraded response]
  RL -- admit --> TO[Timeout budget]
  TO --> CB{Circuit<br/>breaker}
  CB -- open --> FB[Fallback:<br/>cached / default]
  CB -- closed --> BH[Bulkhead pool<br/>per dependency]
  BH --> RT[Retry with<br/>backoff + jitter]
  RT --> DEP[(Dependency)]

Retry storms and why jitter matters

When a dependency recovers, thousands of clients retrying on the same schedule create synchronized spikes that knock it over again. Exponential backoff spreads attempts out; jitter (randomizing the delay) desynchronizes clients. Also cap retries at every layer: three layers each retrying three times means 27 calls to the bottom service.

Real-world example

A product page calls pricing, inventory, reviews and recommendations. Pricing and inventory are essential; reviews and recommendations are not. With 300 ms total budget, each optional call gets a 100 ms timeout and its own bulkhead. If recommendations fail, the circuit opens and the page renders without that carousel. Under extreme load, the edge sheds anonymous bot traffic first, then disables recommendations, protecting checkout.

Advantages and disadvantages

Advantages Disadvantages
Prevents cascading failures Misconfigured retries amplify outages
Keeps core journeys working Fallbacks can hide real problems if not monitored
Predictable behaviour under overload More configuration to tune and test

Technology options

  • Libraries: Resilience4j (Java), Spring Cloud Circuit Breaker, Polly (.NET), failsafe-go.
  • Infrastructure: Envoy/Istio (retries, timeouts, outlier detection), API gateways (rate limits), AWS App Mesh successor options such as VPC Lattice.
  • Testing: AWS Fault Injection Service, Azure Chaos Studio, Chaos Mesh, LitmusChaos, Gremlin, Toxiproxy.

Java example: Resilience4j with Spring Boot

@Service
public class PricingClient {
    private final RestClient rest;

    public PricingClient(RestClient.Builder builder) {
        this.rest = builder.baseUrl("http://pricing").build();
    }

    @CircuitBreaker(name = "pricing", fallbackMethod = "cachedPrice")
    @Retry(name = "pricing")
    @TimeLimiter(name = "pricing")
    public CompletableFuture<Price> price(String sku) {
        return CompletableFuture.supplyAsync(() ->
            rest.get().uri("/prices/{sku}", sku).retrieve().body(Price.class));
    }

    private CompletableFuture<Price> cachedPrice(String sku, Throwable t) {
        return CompletableFuture.completedFuture(PriceCache.lastKnown(sku));
    }
}
resilience4j:
  circuitbreaker:
    instances:
      pricing:
        slidingWindowSize: 50
        failureRateThreshold: 50
        waitDurationInOpenState: 10s
        permittedNumberOfCallsInHalfOpenState: 5
  retry:
    instances:
      pricing:
        maxAttempts: 3
        waitDuration: 100ms
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2
        enableRandomizedWait: true
  timelimiter:
    instances:
      pricing:
        timeoutDuration: 150ms

[!VERSION] Annotation names and property keys follow Resilience4j 2.x with Spring Boot 3. Check the release notes when upgrading to Spring Boot 4.

Architect-level considerations

[!ARCHITECT] Define a timeout budget for each user journey and allocate it across hops. Decide which dependencies are critical vs optional and specify the degraded behaviour for each in the design.

[!EA] Standardize resilience defaults (timeouts, retry policy, circuit breaker settings) in a shared library or mesh configuration so every team does not reinvent them.

Interview questions and answers

Where should retries live: client, gateway, or mesh?

In one place per hop, ideally closest to the caller that knows whether the operation is idempotent. Avoid retries at multiple layers for the same call. Meshes are good for transport-level retries on idempotent methods; business-level retries belong in the application.

Explain the circuit breaker states.

Closed: calls flow, failures are counted. Open: calls fail fast for a cool-down period. Half-open: a few trial calls are allowed; success closes the breaker, failure reopens it.

What is load shedding and how is it different from rate limiting?

Rate limiting enforces a per-client quota regardless of system health. Load shedding drops work based on the system's current capacity and request priority to keep it healthy, for example rejecting batch traffic when CPU or queue depth crosses a threshold.