Data consistency across services
Article.
Splitting code is easy. Splitting state is the hard part, because the moment
two services own two databases, BEGIN stops working:
@db.transaction
def place_order(order):
orders.insert(order) # ours
inventory.reserve(...) # someone else's database
payments.charge(...) # someone else's databaseThe usual answer is a list of patterns — saga, outbox, idempotency. Those are real and each has its own note. But reaching for them first skips the question that actually decides the outcome.
The mechanism: consistency is a boundary decision
A transaction cannot span a service boundary. So the boundary determines what can be transactional, and everything else follows from where you drew it.
That inverts the usual framing. The question is not “which pattern do I use to keep these two services consistent” — it is “why are these two things in different services if they must change together?”
one service, one database two services
BEGIN no BEGIN spans this
reserve stock saga, compensation, outbox,
charge card idempotent consumers,
COMMIT eventual consistency,
reconciliation
strong consistency, free all of the above, foreverThe right-hand column is permanent complexity. It is worth paying when the boundary is real, and it is pure cost when the boundary was drawn by accident.
Aggregates decide the boundary
The DDD concept that answers this concretely: an aggregate is the unit of transactional consistency — a cluster of objects that must be internally consistent at every commit.
The rules that follow are mechanical:
- Inside one aggregate: strong consistency. One transaction, one database, invariants enforced synchronously.
- Across aggregates: eventual consistency. Events, not transactions.
- One transaction should modify one aggregate. If it modifies two, either they are one aggregate, or the invariant between them does not need to be immediate.
So the design work is finding invariants that must hold immediately. “An order’s total equals the sum of its lines” must — same aggregate. “A customer’s loyalty points reflect their orders” does not need to hold within milliseconds — different aggregate, eventual.
Service boundaries should follow aggregate boundaries. A service that owns half an invariant will spend its life in a saga.
When the boundary is real
Sometimes it genuinely is — a third-party payment provider, a separately-owned service, a system you do not control. Then you need the toolkit, and each piece has a job:
| Problem | Pattern |
|---|---|
| Multi-step, no rollback | saga with compensation |
| Write plus event | transactional outbox |
| Duplicate deliveries | idempotent consumers |
| Client retries | idempotency keys |
They compose into one chain rather than being alternatives: the outbox gets the event out reliably, the saga sequences the steps, idempotency makes every step safe to repeat, and compensation handles the failure.
What eventual consistency looks like to a user
The part that turns an architecture decision into a support ticket.
user clicks Save
↓ write commits to the order service
↓ event published
↓ 200 OK, UI navigates to the list
↓ list reads the projection, not caught up
↓ "where did my order go?"The system is behaving correctly and the user is right to be confused. Three practical answers, in order of preference:
Read your own writes. After a user writes, serve that user’s reads from the authoritative source for a short window, rather than the projection.
if recently_wrote(user_id, within=timedelta(seconds=10)):
return await primary.get_orders(user_id)
return await projection.get_orders(user_id)Return the result, do not re-fetch it. The write already knows what it created; render that rather than navigating to a list that has to catch up.
Make the pending state visible. A row marked “processing” is honest and unsurprising. A row that is simply absent looks like data loss — which is why the semantic lock from a saga doubles as a UI affordance.
Where it breaks in production
Replica lag mistaken for eventual consistency
These are different problems with the same symptom. Replica lag is a database concern with a database fix — route that user’s reads to the primary briefly. Confusing it with a projection lag sends you rewriting the wrong layer. See Replication, sharding and consistency models.
Nothing reconciles
Every pattern above is best-effort under failure, so over months the systems drift: an order marked paid with no payment record, a reservation with no order. A reconciliation job that compares the two sides and reports mismatches is not optional at scale — it is how you find out, and most teams add it after an audit rather than before.
The cheap version is a scheduled query per invariant, and it is worth writing before you need it:
async def find_orphans(window: timedelta) -> list[str]:
"""Orders marked paid with no matching payment, older than the window."""
cutoff = utcnow() - window # ignore in-flight sagas
paid = await orders.ids_where(status="PAID", before=cutoff)
settled = await payments.settled_order_ids(paid)
return sorted(set(paid) - set(settled))Two details make it useful rather than noisy. The window excludes sagas still legitimately in flight, so it reports drift rather than latency — set it above the slowest saga’s deadline. And the output goes to a queue a human reads, not a log line: a mismatch is money or inventory in a wrong state, and auto-repairing it without knowing why it drifted usually makes it worse.
Distributed data, no distributed query
“Show me orders with their customer’s tier” spans two services and there is no join. The answers are a projection maintained by events, an API composition in a gateway, or accepting two round trips — all of which are work that a single database did for free.
Chatty services standing in for a join
A service calling three others per request has made the network part of every query. Latency is the sum, and any one being down takes the endpoint down. This is usually the boundary problem again, presenting as a performance problem.
The trade not taken
The strongest answer to a distributed-consistency question is often to not distribute the data.
A modular monolith — clear module boundaries, enforced dependencies, separate schemas if you like, one database and one transaction — gives you every consistency property for free. You can split later, and the module boundaries tell you where; splitting is a deployment decision that can follow the design rather than precede it.
The cost of getting this wrong is asymmetric. A monolith that should have been split is annoying and refactorable. Services split across an invariant are a saga you maintain forever, and un-splitting them means merging two databases in production.
So the honest interview answer to “how would you keep these consistent” starts with “do they need to be in different services?” — and only then reaches for the patterns.
Related
Interview angle 6
- “How do you keep data consistent across microservices?” — first, by not
needing to: a transaction cannot cross a service boundary, so the boundary
decides what can be transactional. If two things must change together, they
probably belong in one service.
- Follow-up: “And when the boundary is genuinely fixed?” — the chain: an outbox to publish reliably, a saga to sequence the steps with compensation, and idempotent consumers because delivery is at-least-once. They compose; they are not alternatives.
- “How do you decide where the boundary goes?” — by invariant. An aggregate is the unit of transactional consistency, so anything that must hold immediately belongs inside one, and anything that can be eventually consistent can cross. A service owning half an invariant lives in a saga.
- “A user saves something and it is not in the list.” — the projection has not caught up. Serve that user’s reads from the authoritative source for a few seconds after their write, return the created object rather than re-fetching, and make pending states visible — an absent row reads as data loss.
- “How do you know your services have not drifted?” — a reconciliation job comparing both sides and reporting mismatches. Every pattern here is best-effort under failure, so drift is certain over months; the only question is whether you find it or an auditor does.
- “How do you query across services?” — you do not join. A projection built
from events, composition at a gateway, or two round trips. All of it is work a
single database was doing for free, which is worth saying out loud when
someone proposes a split.
- Follow-up: “What is wrong with just calling the other service?” — it puts the network inside every query. Latency becomes the sum of the calls, and any one dependency being down takes the endpoint down with it. A service that fans out to three others per request has usually inherited a boundary problem that is presenting as a performance problem.
- “What would you do instead?” — a modular monolith with enforced module boundaries and one database. Splitting later is a deployment decision the design can inform. Un-splitting services drawn across an invariant means merging databases in production, and the asymmetry is the point.