Replication

Leader-follower, multi-leader and leaderless replication; synchronous vs asynchronous; replication lag, failover, split brain and quorum maths.

What it is

Replication keeps copies of the same data on multiple nodes for availability, durability, read scaling and locality.

Model Writes go to Examples Main risk
Single leader (primary/replica) One leader PostgreSQL, MySQL, Aurora, Cloud SQL, Azure SQL Failover time, replica lag
Multi-leader Several leaders (often one per region) MySQL Group Replication multi-primary, CouchDB, active-active setups Write conflicts
Leaderless (quorum) Any replica; client or coordinator writes to W of N Cassandra, ScyllaDB, Dynamo-style systems Stale reads unless W + R > N
Consensus (Raft/Paxos) Leader elected by majority, log replicated to majority etcd, Spanner, CockroachDB, Kafka KRaft metadata Needs majority available

Synchronous vs asynchronous

  • Synchronous: the write is acknowledged after replicas confirm. No data loss on leader failure (RPO = 0) but higher latency and reduced availability if replicas are slow.
  • Asynchronous: acknowledged by the leader only. Fast, but recent writes can be lost on failover and replicas lag.
  • Semi-synchronous / quorum: wait for some replicas (e.g. one of two, or a majority).

Quorum rule: with N replicas, write quorum W and read quorum R, if W + R > N every read overlaps at least one node holding the latest write (assuming no sloppy quorums). Typical: N = 3, W = 2, R = 2.

flowchart LR
  W[Writer] --> L[(Leader)]
  L -- sync --> F1[(Follower AZ-b)]
  L -- async --> F2[(Follower AZ-c)]
  L -- async, cross-region --> F3[(DR replica region 2)]
  Rd[Readers] --> F1
  Rd --> F2

Failure handling

  • Failover: promote a replica when the leader fails. Must fence the old leader (STONITH, epoch numbers) to avoid split brain, where two nodes both accept writes.
  • Replication lag causes stale reads, read-your-writes violations and non-monotonic reads.
  • Conflict resolution for multi-leader/leaderless: last-writer-wins (data loss risk), version vectors with app merge, CRDTs.

Real-world example

Amazon Aurora replicates storage six ways across three AZs and acknowledges writes after a 4-of-6 quorum, separating durability from compute instances; read replicas share the storage volume (documented in the Aurora SIGMOD 2017 paper). Aurora Global Database then replicates asynchronously to other regions for DR with typically sub-second lag.

Advantages and disadvantages

Advantages Disadvantages
Survives node/AZ failure Consistency issues with async replicas
Scales reads Write amplification and network cost
Data close to users Conflict handling for multi-writer
Enables zero-downtime maintenance Failover complexity and split-brain risk

Architect-level considerations

[!ARCHITECT] Match replication mode to RPO: RPO = 0 requires synchronous or quorum replication within the failure domain you protect against. For cross-region RPO = 0 you need a consensus-based multi-region database and must accept the write latency.

[!DEVELOPER] Treat replicas as possibly stale. For flows that read immediately after writing, read from the primary or use session tokens.

Interview questions and answers

What is split brain and how is it prevented?

Two nodes both believe they are leader and accept conflicting writes, usually after a network partition. Prevent it with majority-based leader election (only a side with quorum can lead), fencing tokens/epochs that storage checks, and shutting down or isolating the old leader.

With N = 5, what W and R give strong reads and tolerate two failures for writes?

W = 3 and R = 3 (W + R = 6 > 5). Writes succeed with two nodes down; reads also succeed with two nodes down.

Why not always use synchronous replication?

Every write pays the slowest replica's latency, and if the synchronous replica is unavailable writes may stall. Many systems use synchronous within a region (cheap latency) and asynchronous across regions.