Sagas and compensation
Article.
An order needs a payment charged, stock reserved and a shipment booked, and each
lives in a different service with its own database. There is no BEGIN that
spans all three.
A saga is the answer: a sequence of local transactions, each with a compensating action that undoes it if a later step fails. The interesting parts are that compensation is not rollback, and that a saga has no isolation — which is the half most people never mention.
The mechanism: why not a distributed transaction
Two-phase commit exists and does work. It is not used across service boundaries because of what it does while it works.
coord A B C
|-- prepare ->| | | locks held
|-- prepare --------->| | locks held
|-- prepare ----------------->| locks held
X <- coordinator dies
A, B, C are now "in doubt": holding locks,
unable to commit, unable to abort.Between prepare and commit every participant holds locks and cannot proceed alone. A coordinator failure leaves them blocked indefinitely — they cannot commit, because someone might have voted no; they cannot abort, because someone might have committed.
So 2PC trades availability for atomicity, and across services owned by different teams, deployed independently, that trade is nearly always wrong.
A saga makes the opposite trade: stay available, give up atomicity, and repair afterwards.
Compensation is not rollback
The distinction that decides your design.
A rollback restores the previous state as though nothing happened. A compensating action is a new business fact that offsets an earlier one, and both remain true.
rollback: charge $100 -> [rollback]
-> nothing ever happened
compensation: charge $100
refund $100
-> the statement shows BOTHThat has consequences you have to design around rather than hide:
- The customer sees a charge and a refund, and may call about it.
- The audit log contains both, correctly.
- Some steps have no compensation at all. You cannot un-send an email; the best available compensation is a second email, which is a different thing.
Which produces the first real design rule: order the steps so that anything irreversible comes last. Reserve stock, charge the card, and only then send the confirmation — because once the email is out, the saga has no way back.
Choreography or orchestration
Choreography
Each service listens for events and emits its own. No coordinator.
OrderPlaced -> [payment]
PaymentTaken -> [stock]
StockReserved -> [shipping]
PaymentFailed -> [order] cancelsNothing owns the workflow, which is its appeal and its problem: no single place knows the state of a saga. Answering “where did order 1042 get to” means reconstructing it from logs across four services, and the flow itself exists only as an emergent property of who happens to subscribe to what.
Fine for two or three steps. Past that it becomes a system nobody can draw.
Orchestration
A coordinator holds the workflow explicitly and calls each step.
async def place_order(order_id: str) -> None:
saga = Saga(order_id)
try:
await saga.step(reserve_stock, compensate=release_stock)
await saga.step(charge_card, compensate=refund_card)
await saga.step(book_shipment, compensate=cancel_shipment)
await saga.step(send_confirmation) # last: no compensation
except StepFailed:
await saga.compensate() # in reverse orderThe workflow is one readable function, the state is in one place, and “where did this get to” is a query rather than an investigation.
The cost is a component that knows about every service — which is real coupling, though it is coupling to a workflow rather than between the services themselves. Orchestration is the right default past three steps, and a workflow engine is the mature version of that coordinator, since durability and retries are exactly what it provides — see Temporal vs Celery vs Other Orchestrators.
| Choreography | Orchestration | |
|---|---|---|
| Workflow lives | nowhere explicit | in one component |
| “What state is it in” | reconstruct from logs | query the coordinator |
| Coupling | to event schemas | to the coordinator |
| Good for | 2-3 steps | anything longer |
The part everyone skips: there is no isolation
A saga gives you A, C and D. It does not give you I.
Between step 2 and step 5, the world can see partial results — stock reserved but not paid for, an order that exists but is not confirmed. Other transactions run against that intermediate state, and unlike a database transaction, nothing hides it.
t0 reserve stock -> 3 units held
t1 charge card -> FAILS
t2 release stock -> 3 units returned
between t0 and t2 another customer saw
"out of stock" and went elsewhere.
Nothing was incorrect. You lost the sale.The countermeasures are design choices, not framework settings:
- Semantic lock. Mark the record with a pending status —
PENDING_PAYMENT— so other readers can see it is in flight and decide for themselves. The most broadly useful one. - Commutative updates. Prefer operations whose order does not matter:
balance = balance - 10composes and compensates cleanly;balance = 90does not. - Reread before acting. A step that assumes a value read three steps ago is assuming isolation it does not have. Re-read inside the step.
Naming this unprompted is the strongest signal in a saga question, because it is the difference between having read about sagas and having run one.
Where it breaks in production
The compensation that fails
The obvious question and the one people have not thought about: what if
refund_card throws?
Compensations must be retried until they succeed, which means they must be idempotent, which means the whole idempotency-key discipline applies to them too — see Idempotency keys.
And they must eventually give up into a queue a human reads. A compensation that cannot complete is money or inventory in a wrong state, and the only honest design admits that some of those end in a person’s inbox.
The saga that stops
A step calls a service that never responds. Without a timeout the saga sits half-applied forever, holding a semantic lock, invisible.
Every step needs a deadline, and the saga needs a total one. Alert on sagas in a non-terminal state older than N minutes — that metric is the one that finds this, and its absence is why these are usually discovered by a customer.
Compensating in the wrong order
Compensations run in reverse. Releasing stock before refunding the card can let another order consume the stock while the money is still with you, which is the same failure the saga was avoiding, arriving during recovery.
Events lost between the database and the broker
A step commits its local transaction and then fails to publish — the saga stalls with no event to drive it. That is the dual-write problem, and the outbox is the answer: write the event in the same transaction as the state change. See The transactional outbox.
The trade not taken
The best saga is the one you did not need.
A saga exists because the operation spans services. If those services share an owner and a deployment, putting the three tables in one database and using one transaction is not a compromise — it is a simpler, stronger system with real isolation and no compensation logic.
That is worth saying plainly in an interview, because “how would you implement a distributed transaction” often has the better answer “I would first check whether it needs to be distributed”. Service boundaries drawn around transactional consistency are boundaries drawn in the wrong place, and a saga is the tax you pay for that decision.
Reach for one when the boundary is genuinely fixed — a third-party payment provider, a separately-owned service, a system you do not control.
Related
Interview angle 7
- “How do you do a transaction across three services?” — you do not. A saga
runs a sequence of local transactions, each with a compensating action, and
gives up atomicity to stay available.
- Follow-up: “Why not two-phase commit?” — because between prepare and commit every participant holds locks, and a coordinator failure leaves them in doubt: unable to commit, unable to abort, blocked indefinitely. That trade is wrong across independently deployed services.
- “What is a compensating action?” — a new business action that offsets an earlier one, not a rollback. A refund does not erase the charge; the customer sees both. And some steps have no compensation at all, which is why irreversible steps go last.
- “Choreography or orchestration?” — choreography for two or three steps, where no coordinator is worth it. Orchestration beyond that, because the workflow becomes readable in one place and “what state is this in” becomes a query rather than an archaeology exercise across four services’ logs.
- “What does a saga not give you?” — isolation. Between the first step and
the last, other transactions see partial state, and nothing hides it.
- Follow-up: “So what do you do about it?” — semantic locks, so a pending status is visible to other readers; commutative operations that compose regardless of order; and re-reading values inside a step rather than trusting one read three steps earlier.
- “What if the compensation fails?” — retry it, which means it must be idempotent, and eventually escalate to a human. A compensation that cannot complete is money or stock in a wrong state, and pretending otherwise is how it stays wrong.
- “How do you find a stuck saga?” — alert on sagas in a non-terminal state older than a threshold. Without that metric they are found by customers.
- “When would you not use one?” — when the services share an owner and a deployment. Three tables in one database with one transaction gives real isolation and no compensation logic. A saga is the cost of a boundary drawn across a transaction.