Replication
Leader-follower, multi-leader and leaderless replication; synchronous vs asynchronous; replication lag, failover, split brain and quorum maths.
What it is
Replication keeps copies of the same data on multiple nodes for availability, durability, read scaling and locality.
| Model | Writes go to | Examples | Main risk |
|---|---|---|---|
| Single leader (primary/replica) | One leader | PostgreSQL, MySQL, Aurora, Cloud SQL, Azure SQL | Failover time, replica lag |
| Multi-leader | Several leaders (often one per region) | MySQL Group Replication multi-primary, CouchDB, active-active setups | Write conflicts |
| Leaderless (quorum) | Any replica; client or coordinator writes to W of N | Cassandra, ScyllaDB, Dynamo-style systems | Stale reads unless W + R > N |
| Consensus (Raft/Paxos) | Leader elected by majority, log replicated to majority | etcd, Spanner, CockroachDB, Kafka KRaft metadata | Needs majority available |
Synchronous vs asynchronous
- Synchronous: the write is acknowledged after replicas confirm. No data loss on leader failure (RPO = 0) but higher latency and reduced availability if replicas are slow.
- Asynchronous: acknowledged by the leader only. Fast, but recent writes can be lost on failover and replicas lag.
- Semi-synchronous / quorum: wait for some replicas (e.g. one of two, or a majority).
Quorum rule: with N replicas, write quorum W and read quorum R, if W + R > N every read overlaps at least one node holding the latest write (assuming no sloppy quorums). Typical: N = 3, W = 2, R = 2.
flowchart LR W[Writer] --> L[(Leader)] L -- sync --> F1[(Follower AZ-b)] L -- async --> F2[(Follower AZ-c)] L -- async, cross-region --> F3[(DR replica region 2)] Rd[Readers] --> F1 Rd --> F2
Failure handling
- Failover: promote a replica when the leader fails. Must fence the old leader (STONITH, epoch numbers) to avoid split brain, where two nodes both accept writes.
- Replication lag causes stale reads, read-your-writes violations and non-monotonic reads.
- Conflict resolution for multi-leader/leaderless: last-writer-wins (data loss risk), version vectors with app merge, CRDTs.
Real-world example
Amazon Aurora replicates storage six ways across three AZs and acknowledges writes after a 4-of-6 quorum, separating durability from compute instances; read replicas share the storage volume (documented in the Aurora SIGMOD 2017 paper). Aurora Global Database then replicates asynchronously to other regions for DR with typically sub-second lag.
Advantages and disadvantages
| Advantages | Disadvantages |
|---|---|
| Survives node/AZ failure | Consistency issues with async replicas |
| Scales reads | Write amplification and network cost |
| Data close to users | Conflict handling for multi-writer |
| Enables zero-downtime maintenance | Failover complexity and split-brain risk |
Architect-level considerations
[!ARCHITECT] Match replication mode to RPO: RPO = 0 requires synchronous or quorum replication within the failure domain you protect against. For cross-region RPO = 0 you need a consensus-based multi-region database and must accept the write latency.
[!DEVELOPER] Treat replicas as possibly stale. For flows that read immediately after writing, read from the primary or use session tokens.
Interview questions and answers
What is split brain and how is it prevented?
Two nodes both believe they are leader and accept conflicting writes, usually after a network partition. Prevent it with majority-based leader election (only a side with quorum can lead), fencing tokens/epochs that storage checks, and shutting down or isolating the old leader.
With N = 5, what W and R give strong reads and tolerate two failures for writes?
W = 3 and R = 3 (W + R = 6 > 5). Writes succeed with two nodes down; reads also succeed with two nodes down.
Why not always use synchronous replication?
Every write pays the slowest replica's latency, and if the synchronous replica is unavailable writes may stall. Many systems use synchronous within a region (cheap latency) and asynchronous across regions.