Reliability and Fault Tolerance

Designing systems that keep producing correct results when components fail, covering failure modes, redundancy, idempotency, durability and the SRE practices that sustain reliability.

What it is

  • Reliability: the probability the system performs its intended function correctly over a period. It includes correctness and data integrity, not just uptime.
  • Fault tolerance: the ability to continue operating correctly when some components fail. A fault is a component deviating from spec; a failure is the system as a whole not providing its service. Fault tolerance stops faults becoming failures.

Failure modes to design for

Failure Example Typical mitigation
Crash Instance terminated, pod OOM-killed Redundancy, health checks, restart
Omission / timeout Network drops packets, dependency hangs Timeouts, retries with backoff, idempotency
Slow (gray failure) Disk degraded, noisy neighbour Latency-based health checks, outlier ejection, hedged requests
Byzantine / corrupt Bad deploy writes wrong data Validation, checksums, canaries, immutable audit trail
Correlated AZ outage, bad config pushed everywhere Cell isolation, staged rollouts, multi-AZ/region
Data loss Disk failure, accidental delete Replication, backups, point-in-time recovery, deletion protection

Key techniques

  • Redundancy (N+1, N+2) and replication.
  • Idempotency: every retried operation must produce the same effect once. Use idempotency keys stored with results.
  • Durable writes: acknowledge only after data is persisted to a quorum or replicated log.
  • Checksums and end-to-end validation of data.
  • Write-ahead logs and transactional outbox so state changes and events never diverge.
  • Error budgets: SRE practice where a missed SLO pauses feature releases to invest in reliability.
sequenceDiagram
  participant C as Client
  participant API as Payment API
  participant DB as Database
  C->>API: POST /payments (Idempotency-Key: k1)
  API->>DB: insert key k1 (unique) + process
  DB-->>API: ok
  API--xC: response lost (timeout)
  C->>API: retry POST /payments (Idempotency-Key: k1)
  API->>DB: lookup k1
  DB-->>API: existing result
  API-->>C: same result, no double charge

Real-world example

Stripe's public API accepts an Idempotency-Key header on POST requests so that a client can safely retry after a network error; the server stores the first result for the key and returns it for retries. This is a documented, widely copied pattern for payment reliability.

Advantages and disadvantages

Advantages Disadvantages
Correct results under failure Extra storage and logic (idempotency, dedup)
Fewer incidents and data fixes Redundancy cost
Enables safe automated retries Testing failure paths takes effort

Technology options

Resilience4j / Polly / Envoy for retries and timeouts; Kafka with idempotent producers and transactions; databases with synchronous replication; Temporal/Step Functions for durable execution; chaos engineering tools for testing.

Architect-level considerations

[!DEVELOPER] Make handlers idempotent, use unique constraints for dedup, never retry non-idempotent calls blindly, and write tests that inject failures.

[!ARCHITECT] Define the system's failure model explicitly in the design document: which failures are tolerated, which cause degradation, which cause an outage, and how data integrity is preserved in each.

[!CTO] Reliability is a product feature with a budget. SLOs and error budgets turn "more reliability" from an argument into a measurable trade-off with feature velocity.

Interview questions and answers

How do you make a message consumer reliable when the queue delivers at least once?

Make the consumer idempotent: record processed message IDs (or a business key) in the same transaction as the side effect, or design operations to be naturally idempotent (set state rather than increment). Use a dead-letter queue for poison messages and alert on it.

What is a gray failure?

A partial failure that health checks do not detect, such as a node that answers health pings but serves requests slowly. Detect it with latency/error-based outlier detection and observe from the client's perspective.

Reliability vs fault tolerance?

Fault tolerance is a means; reliability is the outcome. You add fault tolerance mechanisms (redundancy, retries, replication) to achieve a reliability target.