Service Discovery

In a microservices system, instances come and go (autoscaling, deploys, crashes). Service discovery answers “where is service X right now?” without hard-coding hosts.

The two patterns

Client-side discovery

The client queries a registry, gets a list of healthy instances, and load-balances itself.

client ──> registry (Consul/Eureka/etcd) → returns [10.0.1.5, 10.0.1.6, ...]
client ──> chosen instance directly

• Pros: smart load balancing (latency-aware, sticky), no extra hop.
• Cons: every client needs the registry library; harder polyglot story.

Server-side discovery

The client calls a load balancer or service mesh proxy; the proxy resolves and forwards.

client ──> LB / sidecar ──> registry ──> chosen instance

• Pros: thin clients, language-agnostic. AWS ALB, Kubernetes Service, Envoy sidecar all work this way.
• Cons: extra hop, the LB becomes infra you operate.

Common registry options

On Kubernetes you usually don’t write registry code

A Service is the registry entry. my-svc.default.svc.cluster.local resolves via CoreDNS to a stable ClusterIP; kube-proxy load-balances to ready pods. Readiness probes (not just liveness) gate inclusion.

apiVersion: v1
kind: Service
metadata: { name: orders }
spec:
  selector: { app: orders }
  ports: [{ port: 80, targetPort: 8000 }]

http://orders from another pod in the same namespace works without any client library.

On AWS without K8s

• ECS + Cloud Map — service discovery via Route 53 private zone or HTTP namespace.
• ALB / NLB — server-side LB with target groups; targets registered by ECS/EC2 auto-scaling.

Health checks: liveness vs readiness

• Liveness — “is the process alive?” Failure → kill + restart.
• Readiness — “is it ready to take traffic?” Failure → remove from rotation, do NOT kill.

A service warming up (loading caches, opening DB pool) should report unready until it can serve. Hitting liveness too aggressively before readiness exists is a classic outage cause.

Failure modes

• Stale entries — TTL too long; dead instances still listed. Use TTL ≈ 2× heartbeat interval.
• Split-brain in the registry — Consul/etcd Raft losing quorum stops writes; reads may serve stale data.
• DNS caching at the client — JVMs caching forever is the classic example. Set TTL = 0 or short.
• Cold-start traffic — readiness probe must require the service is actually warm (pool open, migrations complete).

Python client pattern (httpx + retry + jitter)

import httpx, random, asyncio

async def call_orders(payload):
    for attempt in range(3):
        try:
            async with httpx.AsyncClient(timeout=2.0) as c:
                r = await c.post("http://orders/v1/place", json=payload)
                r.raise_for_status()
                return r.json()
        except (httpx.TimeoutException, httpx.HTTPStatusError) as e:
            if attempt == 2:
                raise
            await asyncio.sleep((2 ** attempt) * 0.1 + random.random() * 0.1)

Note: timeout is always set — never default. Default httpx timeout is generous; defaults to None in raw socket calls. A missing timeout is the single most common production hang.