Distributed transactions and sagas

Placing an order touches inventory, payment and the order record. In one database that is one transaction: all three or none. Split the three across services and there is no transaction any more — only three local ones, and every way the second can fail after the first committed. This lesson is about the two answers: the one that tries to keep the transaction (and why nobody wants it), and the one that gives it up deliberately.

#Two-phase commit, and its cost

2PC keeps the atomic guarantee across databases with a coordinator: phase one asks every participant to prepare (do the work, hold the locks, promise you can commit), phase two tells them all to commit or all to abort. Java has had it for twenty-five years as XA and JTA, and application servers built their reputations on it.

The cost is in the word "hold". Between prepare and commit every participant keeps its locks, and if the coordinator dies in that window they keep them until it comes back — a prepared transaction is a lock that no timeout releases, because releasing it would break the promise. So the whole system's availability becomes the coordinator's, the slowest participant sets everyone's latency, and a participant that is not a database — a payment provider's HTTP API, a mail service — cannot prepare at all. That last point is the one that settles it for a service architecture: most of what you would want in the transaction cannot join one.

2PC is the right tool inside a single database's replication, and inside Kafka's own transactions, where the participants are built for it. Between services, it is the design that turns a payment provider's five-minute outage into your order table's five-minute lock.

#A saga: local transactions, plus the way back

A saga replaces the one transaction with a sequence of local ones, each committed on its own, and for each a compensating action that undoes its effect if a later step fails:

T1  reserve inventory          C1  release the reservation
T2  charge the card            C2  refund the charge
T3  create the order           C3  cancel the order

If T2 fails, run C1. If T3 fails, run C2 then C1. The guarantee is weaker than atomicity and it is the guarantee you actually get: every step completes, or every completed step is compensated. In between — after T2 and before T3 — the system is visibly inconsistent: the card is charged and there is no order. A saga does not hide that window; it bounds it and promises to resolve it.

Two consequences follow. Compensation is semantic, not a rollback: a refund is a new transaction the customer can see on their statement, not the absence of a charge, and some actions have no compensation at all (an email sent, a shipment dispatched) — those go last, after everything that can fail. And every step must be idempotent, because the saga will retry it after a crash and cannot know whether the previous attempt landed; the distributed-systems course's idempotency lesson is a prerequisite here, not a nice-to-have.

#Choreography: the steps react to each other

Nobody coordinates. Inventory listens for OrderRequested, reserves, and publishes InventoryReserved; payment listens for that, charges, and publishes PaymentTaken; orders listens for that and creates the order. A failure publishes its own event — PaymentFailed — and inventory listens for that to release.

It is the observer pattern across services, and it has observer's costs at scale: no single place says what the saga is. The order of steps is the graph of who listens to whom, which lives in every service and in none, and the question "why is this order stuck" is answered by reading four codebases. Choreography is right for two or three steps that rarely change; at five steps with branches it is a state machine nobody drew.

#Orchestration: one component runs the script

An orchestrator — a class, a workflow engine, a Spring state machine — holds the saga's definition and drives it: send the reserve command, wait for the reply, send the charge command, and on failure send the compensations in reverse order. The participants are simple (do this, reply); the orchestrator is the one place the sequence exists, which is where you want it.

class OrderSaga {
    void run(OrderRequest req) {
        var reservation = inventory.reserve(req.lines());              // step 1
        try {
            var payment = payments.charge(req.customer(), req.total()); // step 2
            try {
                orders.create(req, reservation, payment);              // step 3
            } catch (Exception e) { payments.refund(payment); throw e; }
        } catch (Exception e) { inventory.release(reservation); throw e; }
    }
}

That is the shape, and its flaw is the reason engines exist: if the orchestrator's process dies between step 2 and step 3, the try is gone and so is the knowledge that a refund is owed. A real orchestrator persists the saga's state after every step — a saga table with id, step, status, or Temporal, Camunda, or Axon doing that for you — so a restart picks up where the crash happened. Persisting the state is the whole difficulty; the sequencing is trivial.

The cost of orchestration is a component that knows about every participant and a temptation to put business rules in it; keep it a sequencer, and keep each step's rules in the service that owns them.

#The saga that gets stuck

Every saga will eventually stop halfway: the payment provider times out, the refund fails because the charge is still pending, the orchestrator restarts and the reply was lost. Design for the stuck state before the first one happens:

• Timeouts per step, and a saga-level deadline after which it is compensated or escalated, never left.
• Retries with backoff on every step, which is only safe because every step is idempotent.
• A status you can query — the persisted state above — and a dashboard or a query that lists sagas older than the deadline. The stuck saga that nobody can see is the one that becomes a support ticket a week later.
• Compensation that can also fail, so a compensation is retried the same way, and a compensation that keeps failing is a human's job: an alert with the saga id, not a silent loop.
• Semantic locks for the window: the reserved inventory is marked PENDING, not removed, so a reader can tell the difference between sold and in-flight.

The outbox lesson next is the mechanism that makes "publish an event in the same step as the local write" reliable, which every saga above quietly assumed.