Reliability and Fault Tolerance
Designing systems that keep producing correct results when components fail, covering failure modes, redundancy, idempotency, durability and the SRE practices that sustain reliability.
What it is
- Reliability: the probability the system performs its intended function correctly over a period. It includes correctness and data integrity, not just uptime.
- Fault tolerance: the ability to continue operating correctly when some components fail. A fault is a component deviating from spec; a failure is the system as a whole not providing its service. Fault tolerance stops faults becoming failures.
Failure modes to design for
| Failure | Example | Typical mitigation |
|---|---|---|
| Crash | Instance terminated, pod OOM-killed | Redundancy, health checks, restart |
| Omission / timeout | Network drops packets, dependency hangs | Timeouts, retries with backoff, idempotency |
| Slow (gray failure) | Disk degraded, noisy neighbour | Latency-based health checks, outlier ejection, hedged requests |
| Byzantine / corrupt | Bad deploy writes wrong data | Validation, checksums, canaries, immutable audit trail |
| Correlated | AZ outage, bad config pushed everywhere | Cell isolation, staged rollouts, multi-AZ/region |
| Data loss | Disk failure, accidental delete | Replication, backups, point-in-time recovery, deletion protection |
Key techniques
- Redundancy (N+1, N+2) and replication.
- Idempotency: every retried operation must produce the same effect once. Use idempotency keys stored with results.
- Durable writes: acknowledge only after data is persisted to a quorum or replicated log.
- Checksums and end-to-end validation of data.
- Write-ahead logs and transactional outbox so state changes and events never diverge.
- Error budgets: SRE practice where a missed SLO pauses feature releases to invest in reliability.
sequenceDiagram participant C as Client participant API as Payment API participant DB as Database C->>API: POST /payments (Idempotency-Key: k1) API->>DB: insert key k1 (unique) + process DB-->>API: ok API--xC: response lost (timeout) C->>API: retry POST /payments (Idempotency-Key: k1) API->>DB: lookup k1 DB-->>API: existing result API-->>C: same result, no double charge
Real-world example
Stripe's public API accepts an Idempotency-Key header on POST requests so that a client can safely retry after a network error; the server stores the first result for the key and returns it for retries. This is a documented, widely copied pattern for payment reliability.
Advantages and disadvantages
| Advantages | Disadvantages |
|---|---|
| Correct results under failure | Extra storage and logic (idempotency, dedup) |
| Fewer incidents and data fixes | Redundancy cost |
| Enables safe automated retries | Testing failure paths takes effort |
Technology options
Resilience4j / Polly / Envoy for retries and timeouts; Kafka with idempotent producers and transactions; databases with synchronous replication; Temporal/Step Functions for durable execution; chaos engineering tools for testing.
Architect-level considerations
[!DEVELOPER] Make handlers idempotent, use unique constraints for dedup, never retry non-idempotent calls blindly, and write tests that inject failures.
[!ARCHITECT] Define the system's failure model explicitly in the design document: which failures are tolerated, which cause degradation, which cause an outage, and how data integrity is preserved in each.
[!CTO] Reliability is a product feature with a budget. SLOs and error budgets turn "more reliability" from an argument into a measurable trade-off with feature velocity.
Interview questions and answers
How do you make a message consumer reliable when the queue delivers at least once?
Make the consumer idempotent: record processed message IDs (or a business key) in the same transaction as the side effect, or design operations to be naturally idempotent (set state rather than increment). Use a dead-letter queue for poison messages and alert on it.
What is a gray failure?
A partial failure that health checks do not detect, such as a node that answers health pings but serves requests slowly. Detect it with latency/error-based outlier detection and observe from the client's perspective.
Reliability vs fault tolerance?
Fault tolerance is a means; reliability is the outcome. You add fault tolerance mechanisms (redundancy, retries, replication) to achieve a reliability target.