Backend / Microservices / 08_event_driven_microservices.md

Event-Driven Microservices

Updated 6 interview angles 5 min read source
On this page10
  1. Why services reach for events
  2. Each service owns its data → eventual consistency
  3. Event-carried state transfer kills the chatty read
  4. The dual-write problem is unavoidable here
  5. Coordinating multi-step flows
  6. Schema is a cross-team contract
  7. Observability is harder — design for it up front
  8. Choosing the backbone
  9. Common pitfalls
  10. Interview angle

Event-Driven Microservices

Microservices that integrate through an asynchronous event backbone instead of synchronous request/response calls. Each service owns its data and reacts to events from others; no service blocks on another to do its job.

For the general event style see Event-Driven Architecture. For sync-vs-async transport choices see Inter-service Communication. For cross-service consistency see Data Consistency Across Services.

Why services reach for events

Synchronous call chains couple services in time — every callee must be up, fast, and reachable for the caller to succeed.

text
# Synchronous chain — fails and slows as a unit
Order → Payment → Inventory → Shipping
  if Inventory is down, the whole request fails
  total latency = sum of every hop
  Order is coupled to all three
text
# Event-driven — each service reacts on its own schedule
Order ──OrderPlaced──► (broker) ──► Payment
                                ──► Inventory
                                ──► Analytics
  a slow/down consumer doesn't fail the producer; events queue and drain later
Property Sync (REST/gRPC) Event-driven
Temporal coupling high — callee must be up low — broker buffers
Failure blast radius cascades up the chain contained per consumer
Latency sum of hops producer returns immediately
Adding a consumer change the caller subscribe, producer untouched
Debuggability a stack/trace needs correlation ids + tracing

Each service owns its data → eventual consistency

The defining constraint: one database per service, no shared tables, no cross-service joins, no distributed ACID transaction. State is synchronized by events, so the system is eventually consistent — there’s a window where services disagree, and the design must tolerate it (pending states, reconciliation, idempotent updates).

Event-carried state transfer kills the chatty read

To avoid calling another service on every request, a service keeps a local read replica of just the data it needs, updated from events.

python
# Shipping keeps its own copy of the customer address,
# fed by CustomerAddressChanged events — no call to Customer service at ship time.
def on_customer_address_changed(evt):
    local_addresses.upsert(evt.customer_id, evt.address)

def ship(order):
    # local, fast, no remote dep
    addr = local_addresses.get(order.customer_id)
    courier.dispatch(order, addr)

This trades storage and eventual staleness for autonomy and latency — usually the right trade in microservices.

The dual-write problem is unavoidable here

Every service that mutates state and emits an event hits it: the DB commit and the publish can’t be one atomic action. Solve with the transactional outbox (write event + data in one local transaction, a relay publishes) or CDC reading the DB log (Debezium). See Transactional Outbox Pattern.

python
def confirm_order(order):
    # one local transaction
    with db.transaction():
        orders.update(order, status="CONFIRMED")
        # not a second remote write
        outbox.insert(OrderConfirmed(order.id))
    # a separate relay reads outbox → publishes → marks sent

Coordinating multi-step flows

When one business operation spans services (place order → charge → reserve → ship), you can’t wrap it in a transaction. Use a saga: a sequence of local transactions with compensating actions to undo earlier steps when a later one fails.

  • Choreography — services react to each other’s events; no coordinator. Good for short flows; logic gets scattered as steps grow.
  • Orchestration — a coordinator (often Temporal / AWS Step Functions) drives steps and issues compensations. Preferred once the flow has several ordered steps.

Full treatment, including the “step N fails, roll back 1..N-1” rollback scenario: Data Consistency Across Services and Event-Driven Architecture and Sagas.

Schema is a cross-team contract

Events cross team boundaries, so the payload is a public API. Use a schema registry (Avro/Protobuf) with compatibility checks; add fields as optional; version the event (OrderPlaced.v2) for breaking changes. A careless payload change breaks every downstream team at once.

Observability is harder — design for it up front

A request that fans out across async consumers has no single stack trace. Thread a correlation/trace id through every event and propagate it into logs and spans, so one business operation can be reconstructed across services. See Distributed Tracing.

Choosing the backbone

Broker Fit
Kafka high-throughput, ordered per partition, durable log, replay for new consumers — the event-sourcing/CDC default
RabbitMQ rich routing, per-consumer queues, task-style flows at lower throughput
AWS SNS+SQS managed; SNS fan-out + SQS per-consumer queue
NATS / Redis Streams lightweight, low-latency pub/sub for medium scale

See What is Kafka and What is RabbitMQ.

Common pitfalls

  • Distributed monolith — services that still call each other synchronously for every operation get microservice overhead with monolith coupling. Events are what break the coupling.
  • Sync read in the hot path — calling another service on every request reintroduces temporal coupling. Replicate the data via events instead.
  • No idempotency — at-least-once delivery means duplicates; non-idempotent consumers double-charge. Dedupe on event id.
  • Naive dual write — DB write then publish without an outbox loses events on partial failure.
  • Choreography sprawl — long event chains with logic spread across services become impossible to follow; orchestrate.
  • Ignoring eventual consistency in the UX — show pending/processing states; don’t promise immediate global consistency you can’t deliver.

Interview angle 6

  • “Why use events between microservices instead of REST?” — to remove temporal coupling: producers don’t block on consumers, failures don’t cascade, consumers scale and deploy independently, and new consumers attach without touching the producer.
  • “Each service has its own DB — how do you keep data consistent?” — you don’t get distributed ACID; you get eventual consistency via events, with sagas + compensating transactions for multi-step operations and the outbox pattern for reliable publishing.
  • “How does a service avoid calling another on every request?” — event-carried state transfer: keep a local read replica of the needed data, updated from the owning service’s events; read locally, accept slight staleness.
  • “What’s the dual-write problem and how do you fix it in a service?” — a service can’t atomically commit its DB and publish an event; use a transactional outbox (or CDC) so the event is written in the same transaction and relayed afterward.
  • “How do you debug a request that spans many services asynchronously?” — propagate a correlation/trace id through every event and into logs/spans (distributed tracing), so the whole flow can be reconstructed without a single call stack.
  • “When is event-driven the wrong choice for services?” — when an operation needs an immediate synchronous answer, when strong consistency is required, or when the team can’t yet operate a broker and handle duplicates/ordering/observability.