Lesson 8.3• 10 min read

Distributed transactions: two-phase commit and sagas

Overcoming Two-Phase Commit blocking hazards with orchestrated vs choreographed compensating sagas.

What you will learn

Explain why two-phase commit blocks when its coordinator fails.
Design a saga from compensable steps, a pivot and retryable steps.
Choose between choreography and orchestration, and counter the saga's lack of isolation.

Two-phase commit and why it blocks

Inside one database, a transaction is atomic across tables. With a database per service, one business operation spans several databases.

The classic protocol for a transaction across databases is two-phase commit (2PC):

Prepare: the coordinator asks every participant to promise that it can commit. Each participant must keep holding its locks until it learns the outcome.
Commit: if every participant voted yes, the coordinator tells them all to commit; otherwise, to roll back.

Gray and Lamport state the core problem in one line: "The classic Two-Phase Commit protocol blocks if the coordinator fails." A participant that has voted yes cannot decide on its own, so its locks stay held until the coordinator comes back. Their Paxos Commit protocol removes the block by running consensus on the decision across $2F + 1$ coordinators, and it makes progress while $F + 1$ of them work. AWS's guidance goes further for microservices: with a database per service, "the two-phase commit is not an option", and it recommends sagas instead.

The saga: a sequence of local transactions

Garcia-Molina and Salem (1987) introduced the saga for long-lived transactions. It is a sequence of transactions $T_1, T_2, \dots, T_n$ , each of which commits on its own, with a compensating transaction $C_i$ for each $T_i$ . If $T_k$ fails, the steps that already committed are undone in reverse:

T_1 \rightarrow \dots \rightarrow T_{k-1} \rightarrow \text{(}T_k\text{ fails)} \rightarrow C_{k-1} \rightarrow \dots \rightarrow C_1

Richardson defines it for microservices as "a reliably coordinated sequence of local transactions". There is no automatic rollback: "a developer must design compensating transactions that explicitly undo changes made earlier in a saga". A compensation undoes a step's effect for the business. It does not restore the old bytes; a payment, for example, is refunded rather than deleted.

Order the steps: compensable, pivot, retryable

The Azure Architecture Center sorts saga steps into three kinds:

Compensable steps can be undone by another transaction with the opposite effect.
The pivot is "the point of no return". Once it succeeds, the saga must run to completion.
Retryable steps come after the pivot. They are idempotent and retried until they succeed, instead of being undone.

A checkout saga shaped this way:

Step	Kind	Compensation
$T_1$ Create order as `PENDING`	Compensable	$C_1$ Set status to `CANCELED`
$T_2$ Reserve stock	Compensable	$C_2$ Release the reservation
$T_3$ Charge the card	Pivot	none needed: if it fails, $C_2$ and $C_1$ run
$T_4$ Create the shipment	Retryable	none: retried until it succeeds

Reserving stock before charging means an out-of-stock failure never needs a refund.

Choreography and orchestration

Choreography: "each local transaction publishes domain events that trigger local transactions in other services" (Richardson). There is no central coordinator.
Orchestration: "an orchestrator (object) tells the participants what local transactions to execute". AWS's guidance shows AWS Step Functions in this role.

	Choreography	Orchestration
Suits	Simple workflows with few services	Complex workflows, or ones that gain new steps
Single point of failure	None; responsibility is spread across participants	The orchestrator, unless it is itself highly available
Dependencies	Risk of cyclic dependencies between participants	Avoided, because the orchestrator owns the flow
Understanding the flow	Hard to track which events each participant responds to	Flow is in one place; coordination logic must be written

(Summarized from the Azure Architecture Center and AWS Prescriptive Guidance.)

Either way, each participant must update its database and publish its message atomically (the transactional outbox, chapter 7). It must also be idempotent, because messages can be redelivered.

Lack of isolation

Each step commits at once, so other transactions can see a saga's intermediate states. Azure lists the resulting anomalies as lost updates, dirty reads and fuzzy (non-repeatable) reads. The countermeasures include:

Semantic lock: an application-level flag such as status = 'PENDING' tells other transactions the record is mid-saga. Richardson calls the PENDING state "a semantic lock counter-measure".
Commutative updates: design updates so they give the same result in any order.
Pessimistic view: reorder the saga so updates happen in retryable steps.
Reread values: check the data has not changed before updating it.

Compensations can fail too, which can leave the system inconsistent (Azure), so they need retries and monitoring of their own.

Checkout saga

Local transactions in order; on failure, compensations run in reverse. Steps after the pivot are retried, not undone.

Coordination

Failure

T1 · CompensableOrderServiceCreateOrderC1: Set order status to 'CANCELED'Not started
T2 · CompensableInventoryServiceReserveStockC2: Release the reserved unitsNot started
T3 · PivotPaymentServiceChargeCardPoint of no returnNot started
T4 · RetryableShippingServiceCreateShipmentRetried until it succeedsNot started

Outcome: not run yet

Pick a failure and run the saga.

Figure 3: Orchestrated Saga workflow with backward compensating transactions rolling back payment and reserving inventory on failure.

Key takeaways

In two-phase commit a participant that voted yes holds its locks until the coordinator decides, so a failed coordinator blocks it.
A saga commits each step locally and undoes earlier steps with hand-written compensations; after the pivot, steps are retried instead of undone.
Choreography suits a few services and has no coordinator; orchestration keeps a complex flow in one place.
Others can see a saga's intermediate states, so use semantic locks such as a PENDING status, commutative updates or rereads, and monitor compensations, which can fail too.

Sources

Adapted from The System Design Primer by Donne Martin, licensed under CC BY 4.0. Changes were made.