Distributed Transactions
Two-phase commit, sagas, the transactional outbox, idempotent consumers and TCC, with guidance on when each is appropriate and how to avoid needing them.
What it is
A distributed transaction updates data in more than one independent resource (databases, services, queues) and must leave all of them consistent. In a monolith with one database, a local ACID transaction does this. In microservices each service owns its data, so you need a different mechanism.
Options
| Approach | How it works | Consistency | Use when |
|---|---|---|---|
| Two-phase commit (2PC / XA) | Coordinator asks all participants to prepare, then commit | Atomic, strong | Few participants, same trust domain, low latency, supported resources |
| Saga (orchestrated) | Sequence of local transactions; a coordinator triggers compensations on failure | Eventual, with business-level atomicity | Long-running business processes across services |
| Saga (choreographed) | Services react to each other's events | Eventual | Few steps, simple flows |
| Transactional outbox | Write business change + event row in one local transaction; relay publishes event | Atomic between DB and message | Any service that must publish events reliably |
| TCC (Try-Confirm-Cancel) | Reserve resources (try), then confirm or cancel | Eventual with reservations | Inventory, bookings, payments holds |
| Distributed SQL | Database itself provides distributed ACID (Spanner, CockroachDB) | Strong | Data can live in one logical DB |
Why 2PC is avoided across services
2PC blocks: if the coordinator fails after participants voted "prepared", they hold locks until it recovers. It couples availability of all participants, many cloud services and message brokers do not support XA, and latency adds up. It remains valid inside one database engine or between a few XA-capable resources.
Saga example: place order
sequenceDiagram participant O as Order service (orchestrator) participant I as Inventory participant P as Payment participant S as Shipping O->>I: Reserve items I-->>O: Reserved O->>P: Charge card P-->>O: Declined O->>I: Compensate: release reservation I-->>O: Released O->>O: Mark order FAILED, notify customer Note over O,S: On success: P approved → S create shipment → order CONFIRMED
Transactional outbox
BEGIN;
INSERT INTO orders (id, customer_id, status, total) VALUES ($1, $2, 'PENDING', $3);
INSERT INTO outbox (id, aggregate_id, type, payload, created_at)
VALUES (gen_random_uuid(), $1, 'OrderPlaced', $4::jsonb, now());
COMMIT;
-- A relay (Debezium CDC or a poller) publishes outbox rows to Kafka and marks them sent.Consumers must be idempotent because the relay can publish a row more than once.
Real-world example
An online travel agency books a flight, hotel and car from different suppliers. No supplier supports 2PC. The booking orchestrator runs a saga: hold flight → hold hotel → hold car → charge → confirm all; if the car fails, it cancels the hotel and flight holds. Compensation is a business action (cancel booking with possible fee), not a database rollback.
Advantages and disadvantages
| Saga | 2PC |
|---|---|
| No distributed locks; services stay available | Simple atomic semantics |
| Fits long-running, cross-organization flows | Blocking on coordinator failure |
| Intermediate states are visible (need semantic locks like "PENDING") | Poor fit for HTTP services and many cloud brokers |
| Compensations must be designed and tested | Couples participant availability |
Technology options
Orchestrators: Temporal, Camunda/Zeebe, AWS Step Functions, Azure Durable Functions, Google Workflows, Conductor. Outbox/CDC: Debezium, Kafka Connect, DynamoDB Streams, Cosmos DB change feed, Spanner change streams. Java: Spring Modulith event publication registry, Axon Framework.
Architect-level considerations
[!ARCHITECT] First try to avoid the distributed transaction by redrawing boundaries: if two pieces of data must always change together atomically, they probably belong in the same service and database.
[!DEVELOPER] Every saga step and compensation must be idempotent and retryable. Model explicit states (PENDING, CONFIRMED, CANCELLED) rather than relying on rollback.
Interview questions and answers
How do you guarantee an event is published if and only if the database commit succeeds?
Transactional outbox: write the event to an outbox table in the same local transaction, then publish asynchronously via CDC or a poller, and make consumers idempotent. Dual writes (DB then broker) without an outbox can lose or invent events.
Orchestration or choreography?
Orchestration for flows with many steps, branching or compliance visibility needs; it gives a single place to see state. Choreography for simple, loosely coupled reactions; it can become hard to trace as flows grow.
What is a semantic lock?
An application-level flag (such as order status PENDING) that tells other operations the record is in the middle of a saga, so they either wait, reject or handle it accordingly.