Distributed Transactions

Two-phase commit, sagas, the transactional outbox, idempotent consumers and TCC, with guidance on when each is appropriate and how to avoid needing them.

What it is

A distributed transaction updates data in more than one independent resource (databases, services, queues) and must leave all of them consistent. In a monolith with one database, a local ACID transaction does this. In microservices each service owns its data, so you need a different mechanism.

Options

Approach How it works Consistency Use when
Two-phase commit (2PC / XA) Coordinator asks all participants to prepare, then commit Atomic, strong Few participants, same trust domain, low latency, supported resources
Saga (orchestrated) Sequence of local transactions; a coordinator triggers compensations on failure Eventual, with business-level atomicity Long-running business processes across services
Saga (choreographed) Services react to each other's events Eventual Few steps, simple flows
Transactional outbox Write business change + event row in one local transaction; relay publishes event Atomic between DB and message Any service that must publish events reliably
TCC (Try-Confirm-Cancel) Reserve resources (try), then confirm or cancel Eventual with reservations Inventory, bookings, payments holds
Distributed SQL Database itself provides distributed ACID (Spanner, CockroachDB) Strong Data can live in one logical DB

Why 2PC is avoided across services

2PC blocks: if the coordinator fails after participants voted "prepared", they hold locks until it recovers. It couples availability of all participants, many cloud services and message brokers do not support XA, and latency adds up. It remains valid inside one database engine or between a few XA-capable resources.

Saga example: place order

sequenceDiagram
  participant O as Order service (orchestrator)
  participant I as Inventory
  participant P as Payment
  participant S as Shipping
  O->>I: Reserve items
  I-->>O: Reserved
  O->>P: Charge card
  P-->>O: Declined
  O->>I: Compensate: release reservation
  I-->>O: Released
  O->>O: Mark order FAILED, notify customer
  Note over O,S: On success: P approved → S create shipment → order CONFIRMED

Transactional outbox

BEGIN;
  INSERT INTO orders (id, customer_id, status, total) VALUES ($1, $2, 'PENDING', $3);
  INSERT INTO outbox (id, aggregate_id, type, payload, created_at)
    VALUES (gen_random_uuid(), $1, 'OrderPlaced', $4::jsonb, now());
COMMIT;
-- A relay (Debezium CDC or a poller) publishes outbox rows to Kafka and marks them sent.

Consumers must be idempotent because the relay can publish a row more than once.

Real-world example

An online travel agency books a flight, hotel and car from different suppliers. No supplier supports 2PC. The booking orchestrator runs a saga: hold flight → hold hotel → hold car → charge → confirm all; if the car fails, it cancels the hotel and flight holds. Compensation is a business action (cancel booking with possible fee), not a database rollback.

Advantages and disadvantages

Saga 2PC
No distributed locks; services stay available Simple atomic semantics
Fits long-running, cross-organization flows Blocking on coordinator failure
Intermediate states are visible (need semantic locks like "PENDING") Poor fit for HTTP services and many cloud brokers
Compensations must be designed and tested Couples participant availability

Technology options

Orchestrators: Temporal, Camunda/Zeebe, AWS Step Functions, Azure Durable Functions, Google Workflows, Conductor. Outbox/CDC: Debezium, Kafka Connect, DynamoDB Streams, Cosmos DB change feed, Spanner change streams. Java: Spring Modulith event publication registry, Axon Framework.

Architect-level considerations

[!ARCHITECT] First try to avoid the distributed transaction by redrawing boundaries: if two pieces of data must always change together atomically, they probably belong in the same service and database.

[!DEVELOPER] Every saga step and compensation must be idempotent and retryable. Model explicit states (PENDING, CONFIRMED, CANCELLED) rather than relying on rollback.

Interview questions and answers

How do you guarantee an event is published if and only if the database commit succeeds?

Transactional outbox: write the event to an outbox table in the same local transaction, then publish asynchronously via CDC or a poller, and make consumers idempotent. Dual writes (DB then broker) without an outbox can lose or invent events.

Orchestration or choreography?

Orchestration for flows with many steps, branching or compliance visibility needs; it gives a single place to see state. Choreography for simple, loosely coupled reactions; it can become hard to trace as flows grow.

What is a semantic lock?

An application-level flag (such as order status PENDING) that tells other operations the record is in the middle of a saga, so they either wait, reject or handle it accordingly.