AI & ML / Inference & serving / 06_llm_gateway_and_caching.md

The LLM gateway and caching

Updated 6 interview angles 5 min read source
On this page6
  1. What it buys you
  2. Caching: three kinds, increasingly clever
  3. Invalidation is the hard half
  4. Measuring whether it helps
  5. Related
  6. Interview angle

The LLM gateway and caching

Once more than one service calls a model, the same four concerns appear in each of them: keys, routing, retries, and cost tracking. A gateway is the layer that owns them once.

text 
services ──▶ gateway ──▶ provider A
                     ├─▶ provider B
                     └─▶ self-hosted
                     │
             keys, routing, retries,
             caching, budgets, logging

The interview framing: it is the same argument as an API gateway or a service mesh. Cross-cutting concerns belong in one place, not copy-pasted into every service that happens to call a model.

What it buys you

Concern Without With
Provider keys in every service one place, rotated once
Fallback per service, if at all uniform
Cost attribution guesswork per key, per tenant
Model switch a deploy per service a config change
Rate limits each service guesses enforced centrally

Cost attribution is usually the reason it gets built. Without a gateway, “which team spent that?” has no answer, because the provider bill is one number per organisation.

LiteLLM is the common open-source choice — a proxy speaking the OpenAI-compatible shape to a hundred providers, with budgets, keys and logging. Cloud equivalents exist, and rolling a thin one yourself is defensible when you only need two providers.

Gotcha: the gateway is now on the critical path of every AI request. It needs its own availability story, and it must not add a second retry layer on top of the SDK’s — nested retries turn one slow request into nine.

Caching: three kinds, increasingly clever

Exact-match cache

python 
key = sha256(f"{model}:{prompt}".encode()).hexdigest()
if hit := await redis.get(key):
    return hit

Trivial and surprisingly effective for classification, extraction and templated tasks where inputs genuinely repeat. Zero risk of a wrong answer, because the input is identical.

Include everything that changes the output in the key — model, prompt, temperature, tools, schema version. A cache keyed only on the user’s text will serve answers from the previous prompt version after a deploy.

Provider prompt caching

Providers cache the prefix of a prompt: a long system prompt or retrieved context that repeats across calls is billed at a large discount and processes faster.

The requirement is structural — the stable content must come first, because the cache matches a prefix. Putting the user’s message before the system instructions defeats it entirely. That single ordering rule is worth knowing.

Semantic cache

Embed the query, look for a near neighbour above a similarity threshold, and serve its answer.

python 
hits = await index.search(embed(query), k=1)
if hits and hits[0].score > 0.95:
    return hits[0].answer

This is the one to be careful about. “Similar question” is not “same question”. “Can I cancel my order?” and “Can I cancel my subscription?” embed closely and have different answers, so a threshold that is too loose serves confidently wrong responses that are very hard to debug — the trace shows a cache hit, not a bad generation.

Where it works: high-volume FAQ traffic with a narrow domain, a conservative threshold, and per-tenant namespacing so one customer never sees another’s cached answer. Where it does not: anything personalised, anything where the answer depends on state, and anything where being wrong is expensive.

Invalidation is the hard half

A cached RAG answer goes stale when the underlying document changes, and nothing about the query tells you that.

The workable approaches, in order of cost:

  1. Short TTL. Crude, and usually correct. Minutes to hours.
  2. Version the corpus. Include a corpus or index version in the cache key, so a reindex invalidates everything at once.
  3. Tag by source document. Cache entries record which chunks they used; updating a document evicts the entries that cited it. Precise, and the most work.

Option two is the sweet spot for most RAG systems: one counter to bump.

Measuring whether it helps

Track hit rate and the quality of hits, not hit rate alone. A semantic cache with a 60% hit rate and a 5% wrong-answer rate is worse than no cache, and only the second number tells you.

Sample cached responses into your eval set the same way you would sample generations. See Online evaluation and experiments.

Interview angle 6

  • “Why put a gateway in front of the models?” - the same reason as an API gateway: keys, routing, retries, budgets and logging are cross-cutting, so they belong in one place rather than copy-pasted into every service. Cost attribution is usually the trigger, because the provider bill is one number per organisation.
  • “What’s the risk of adding one?” - it is on the critical path of every AI request, so it needs its own availability story, and it must not stack a second retry layer on the SDK’s. Nested retries turn one slow request into nine.
  • “What can you cache?” - exact matches, keyed on everything that changes the output including the prompt version. Provider prompt caching for the stable prefix, which requires the stable content to come first because the cache matches a prefix. And semantically, with care.
  • “What’s wrong with semantic caching?” - similar is not the same. “Cancel my order” and “cancel my subscription” embed closely and have different answers, so a loose threshold serves confidently wrong responses whose traces show a cache hit rather than a bad generation. Use it for narrow high-volume FAQ traffic with a conservative threshold and per-tenant namespaces.
  • “How do you invalidate a cached RAG answer?” - a short TTL usually, or a corpus version in the cache key so a reindex invalidates everything at once. Tagging entries by the chunks they cited is precise and the most work.
  • “How do you know the cache is helping?” - hit rate alone is not enough. A 60% hit rate with a 5% wrong-answer rate is worse than no cache, so sample cached responses into the eval set like any other output.
Previous
Which model for which task