Microservices in production

Article.

You run fifteen services. What does it take for one user request to stay correct across them, and for any one service to deploy without the others? That is the microservices question once the split has happened.

Whether to split at all, the distributed monolith, and what a network boundary costs are in Monolith vs microservices; consistency across services (aggregates, sagas, reconciliation) is in Data consistency across services.

This note is the operating layer in between: a request crossing services (discovery, deadlines, retries), identity across hops, events that are published exactly when the data commits, deploys under version skew, shutting a pod down without errors, and traces that survive queues.

A request crossing services

client → gateway → orders → payments → bank
                      └──▶ outbox → broker → email

Finding each other

On Kubernetes, http://payments.prod.svc.cluster.local resolves through CoreDNS to a Service, and only pods whose readiness probe passes are endpoints. Discovery is mostly free; the part that is not is what readiness means.

A readiness probe that checks the database marks every pod unready together when the database blips, and the Service has no endpoints at all. Readiness should mean “this process can serve”, with dependency failures handled per request. Registries and client-side balancing: Service Discovery.

One deadline, not one timeout per hop

Each service configured with its own 30-second timeout means a user who gave up after 5 seconds still has work running three services deep. Pass the remaining budget down instead:

DEADLINE_HEADER = "x-request-deadline"          # absolute, epoch ms

async def call_payments(request, body):
    deadline = float(request.headers.get(DEADLINE_HEADER, time.time() * 1000 + 5000))
    remaining = deadline / 1000 - time.time() - 0.050   # keep 50 ms to answer
    if remaining <= 0:
        raise HTTPException(504, "deadline exceeded before calling payments")
    return await client.post(
        PAYMENTS_URL, json=body, timeout=remaining,
        headers={DEADLINE_HEADER: str(deadline)},
    )

gRPC does this natively: a client deadline travels as grpc-timeout and a server can pass the context on, so downstream calls inherit what is left. An absolute deadline assumes clocks agree to within a few milliseconds; sending the remaining milliseconds instead avoids that and loses the time spent in transit.

Retries multiply

If every tier retries a failing call up to three attempts, the bottom service sees the product:

1 tier(s) retrying x3 -> up to  3 calls to the bottom service per user request
2 tier(s) retrying x3 -> up to  9 calls to the bottom service per user request
3 tier(s) retrying x3 -> up to 27 calls to the bottom service per user request
4 tier(s) retrying x3 -> up to 81 calls to the bottom service per user request

A dependency that is struggling at 1x load receives 27x and never recovers. The rules: retry at one layer only (usually the one closest to the failure), cap retries with a budget (for example 10% of requests), never retry once the deadline has passed, and make retried operations idempotent. And every synchronous hop multiplies availability too:

1 hops at 99.9% each -> 99.90% (43 min/month of failures)
3 hops at 99.9% each -> 99.70% (129 min/month of failures)
5 hops at 99.9% each -> 99.50% (216 min/month of failures)
8 hops at 99.9% each -> 99.20% (344 min/month of failures)

That is the argument for moving side effects off the request path onto events. Breakers, bulkheads and shedding: Resilience patterns for service-to-service calls.

Identity across hops

The common design: the gateway validates the user’s JWT, then forwards X-User-Id: 42 and services trust it. That is safe only if nothing but the gateway can reach the services. Any other path in (a pod with a public IP, a debugging port-forward left open, an SSRF bug in one service) lets anyone set the header and become user 42.

Two sturdier forms:

• Forward the token and verify it in every service (signature against cached JWKS, issuer, expiry, scope). Each service then authorises against claims it has verified itself. Cost: microseconds per request once the keys are cached.
• Exchange it at the gateway for a short-lived internal token, signed by your own issuer and scoped to the call (OAuth 2.0 Token Exchange, RFC 8693), so a stolen internal token is worth little.

Service identity is a separate question: which service is calling. mTLS with per-service certificates (a mesh issues and rotates them) or workload identity (Kubernetes service accounts, IAM roles for pods) answers it, and policy then says “orders may call payments POST /charges”. Authorisation still happens in the service that owns the data: the gateway cannot know whether user 42 owns order 9.

Events that match the data

The dual write, and the outbox

Writing to the database and then publishing to the broker is two systems with no shared transaction: a crash between them leaves an order with no event, or an event with no order. The transactional outbox writes the event into a table in the same transaction:

async with conn.transaction():
    order = await conn.fetchrow(INSERT_ORDER, ...)
    await conn.execute(
        "INSERT INTO outbox (topic, payload) VALUES ($1, $2)",
        "orders.created", dumps(order),
    )
# Both rows commit, or neither. No broker call in here.

A relay publishes pending rows. FOR UPDATE SKIP LOCKED lets several relays run side by side, but only inside a transaction: on its own the SELECT autocommits and its row locks are gone before the loop starts.

async with conn.transaction():
    rows = await conn.fetch(
        "SELECT * FROM outbox WHERE published_at IS NULL"
        " ORDER BY id FOR UPDATE SKIP LOCKED LIMIT 100"
    )
    for r in rows:
        await producer.send(r["topic"], r["payload"])
        await conn.execute("UPDATE outbox SET published_at = now()"
                           " WHERE id = $1", r["id"])

Publishing before marking is deliberate: a crash between the two republishes the row, so delivery is at least once. Change data capture (Debezium reading the WAL) is the alternative with no relay code and more infrastructure. Trade-offs, including holding the transaction across the publish: The transactional outbox.

Consumers must be idempotent

At-least-once delivery means every consumer sees duplicates: relay republishes, broker redeliveries after a consumer crash, rebalances. Record the message id in the same transaction as the side effect:

BEGIN;
INSERT INTO processed_messages (consumer, message_id)
VALUES ('email', :id)
ON CONFLICT DO NOTHING;            -- 0 rows → already handled, skip
-- ... the side effect's own writes ...
COMMIT;

A side effect outside the database (an email, a card charge) cannot share that transaction; it needs the provider’s idempotency key, or a “sending” state written first and reconciled later. Sagas for multi-step flows: Data consistency across services.

Deploying one service alone

Independent deploys are the point of the architecture, and they mean version skew is normal: during a rolling deploy, old and new versions of a service run side by side, and its callers and consumers are at whatever version they were last deployed at.

Expand, migrate, contract

Every change to an API or event schema goes out in steps that are each compatible with the step before:

1. Expand: add the new field as optional; producers write both old and new; consumers read either (the tolerant reader: ignore unknown fields, default missing ones).
2. Migrate: move consumers to the new field, one deploy at a time; backfill stored data.
3. Contract: once nothing reads the old field (check, do not assume: logs or metrics on its use), stop writing it.

Renaming amount to amount_cents in one release breaks every consumer that has not deployed yet, and every event already sitting in a queue. Events outlive deploys, so event schemas need the stricter discipline.

Checking it in CI

• Consumer-driven contract tests (Pact): each consumer publishes the requests and fields it relies on; the provider’s CI replays them against the new build. A breaking change fails the provider’s build, not production.
• Schema registry for events (Avro or Protobuf with Confluent or Apicurio): the registry rejects a new schema version that breaks the configured compatibility mode (Confluent’s default is BACKWARD).
• End-to-end tests only for a few critical journeys; a shared environment that every change must pass through is how a fleet turns back into a release train.

Shutting a pod down without errors

Deleting a pod starts two things in parallel: the kubelet sends the container SIGTERM, and the control plane removes the pod from the Service’s endpoints, which every node’s kube-proxy and every ingress must then learn about. If the process exits immediately, requests routed in the next second or two hit a closed port:

t=0     pod deleted
t=0     SIGTERM → app exits at once      ✗ too early
t=0-2s  endpoints updated on nodes/LB    ← still routing

The fix is to keep serving until routing has caught up, then drain:

spec:
  terminationGracePeriodSeconds: 45     # covers preStop + drain
  containers:
    - name: api
      lifecycle:
        preStop:
          sleep: {seconds: 10}          # built-in action, Kubernetes 1.30+
      command: ["uvicorn", "app:app", "--timeout-graceful-shutdown", "25"]

SIGTERM arrives after the preStop sleep; uvicorn then stops accepting, finishes in-flight requests for up to 25 s, and runs the lifespan shutdown (close pools, flush telemetry). Workers consuming a queue do the same: stop fetching, finish or release the current message. Older clusters use exec: {command: ["sleep", "10"]}, which needs a sleep binary in the image.

Traces across services and queues

The OpenTelemetry instrumentations put the W3C traceparent header on outbound HTTP calls and adopt it on inbound ones. Traces break wherever the transport is not instrumented: a custom queue, a thread pool, a subprocess. Doing it by hand for a message:

from opentelemetry.propagate import inject, extract

carrier = {}
inject(carrier)                         # {"traceparent": "00-4bf9...-01"}
await producer.send(                    # aiokafka: header values are bytes
    "orders.created", value=body,
    headers=[(k, v.encode()) for k, v in carrier.items()],
)

# consumer
ctx = extract({k: v.decode() for k, v in msg.headers})
with tracer.start_as_current_span("handle order.created", context=ctx):
    ...

contextvars carry the context across await, but not into a plain threading.Thread; use contextvars.copy_context().run(fn). Details: Distributed Tracing.

Gateway and mesh

Both together is normal. A mesh’s retries count toward the amplification above: configure them in one place, the mesh or the client library, not both. More in API Gateway and Service Mesh.

Where it breaks in production

• A retry storm: one slow dependency, retries at three tiers, 27x load, no recovery until someone turns retries off.
• Work after the user left: no deadline propagation, so a cancelled request keeps three services busy for 30 seconds each.
• 502s on every deploy: no preStop delay, so pods exit before the load balancer stops sending them traffic.
• A consumer crash loop on deploy day: a producer renamed a field, and old messages and old consumers disagree with the new schema.
• Orders with no events: publishing straight to the broker after commit, and a crash in between.
• Duplicate emails: a consumer without a processed-message record, after a rebalance redelivered a batch.
• Header spoofing: services trusting X-User-Id, reachable by a path that does not go through the gateway.
• A readiness probe on the database: one database blip removes every pod from every Service at once.

The trade not taken

A service mesh instead of libraries for timeouts, retries, mTLS and telemetry. The mesh gives the same policy in every language without code changes, rotates certificates for you, and is often the cheapest way to get mTLS for compliance.

It costs a proxy per pod (memory, a millisecond or so per hop; sidecar-less modes such as Istio ambient reduce that), a control plane to operate and upgrade, and a second place where retries and timeouts live.

Under about ten services in one language, a shared HTTP client with deadlines, retry budgets and OpenTelemetry built in does the job with less to run.

Related

• Monolith vs microservices - whether to split, and the distributed monolith
• Data consistency across services - aggregates, sagas, reconciliation
• Resilience patterns for service-to-service calls - timeouts, breakers, bulkheads, shedding
• The transactional outbox - the outbox in depth
• Inter-service Communication - sync vs async, REST vs gRPC
• Distributed Tracing - propagation in depth
• Internal SDKs for inter-service calls - the shared client that carries these rules