Outbound auth, logging and caching

The three cross-cutting concerns of calling someone else’s API. All three belong in one place — the client — because scattered across service methods they cannot be reasoned about. The client’s structure itself is Writing a client for an external API.

Token caching is a concurrency problem

The naive version fetches a token per request, which is slow and will get you rate-limited on the token endpoint. The next version caches it and, under load, has every concurrent request notice the expiry at the same instant and refresh simultaneously — the classic stampede.

class TokenCache:
    def __init__(self, fetch):
        self._fetch, self._lock = fetch, asyncio.Lock()
        self._token, self._expires = None, 0.0

    async def get(self) -> str:
        if self._token and time.monotonic() < self._expires:
            return self._token
        async with self._lock:
            # Re-check: another task may have refreshed
            # while we waited for the lock.
            if self._token and time.monotonic() < self._expires:
                return self._token
            tok, ttl = await self._fetch()
            self._token = tok
            self._expires = time.monotonic() + ttl - 60
            return self._token

Three details carry the answer: the double check inside the lock, so only one task refreshes; time.monotonic rather than time.time, so an NTP step cannot make a live token look expired; and the 60-second margin, so you refresh before expiry rather than after a 401.

Rule of thumb: on a 401 mid-flight, refresh once and retry once. Looping on 401 means the credential is genuinely wrong, and repeated attempts trigger lockout rather than recovery.

Where the credential itself lives: a secrets manager fetched at runtime, or workload identity where the platform issues a short-lived credential and there is no stored secret at all. See Secrets and configuration.

Logging: at the boundary, redacted, correlated

async def log_response(response):
    await response.aread()
    log.info(
        "outbound",
        extra={
            "host": response.request.url.host,
            # not the query
            "path": response.request.url.path,
            "status": response.status_code,
            "ms": response.elapsed.total_seconds() * 1000,
            "trace_id": trace_id_var.get(),
        },
    )

client = httpx.AsyncClient(
    event_hooks={"response": [log_response]},
)

An event hook means every call is logged whether or not the caller remembered. Note what is absent: no body, no headers, and the path without the query string — tokens and identifiers travel in query parameters more often than people expect.

Attach the same trace_id your inbound request carries, and the outbound call appears in the same trace as the request that caused it. See Observability.

Caching: honour their headers first

If the API sends ETag or Cache-Control, the cheapest correct cache is a conditional request — you still make the call, and a 304 costs no body:

headers = {}
if (hit := await cache.get(key)):
    headers["If-None-Match"] = hit.etag

r = await client.get(url, headers=headers)
if r.status_code == 304:
    return hit.value                      # still fresh
await cache.set(key, Entry(r.json(), r.headers.get("etag")))

Application-level caching, when they send nothing useful:

Key on method, URL and auth scope. A cache keyed on URL alone serves one tenant’s data to another, which is the single worst bug in this note.

key = f"{method}:{url}:{tenant_id}"