Design an agent platform
The design question implied by every ad that says “own agent evaluation, cost and reliability in production”. Not “build an agent” — build the thing several teams build agents on.
The brief: an internal platform where product teams ship LLM agents. Multiple tenants, regulated industry, agents call internal APIs and take actions with consequences.
Clarify first
The questions that change the design, asked before drawing anything:
- Who are the tenants — internal teams, or customers? Decides whether isolation is a convenience or a security boundary.
- What can agents do — read-only, or write? A platform whose agents only read is a different risk category.
- Latency shape — interactive, or background jobs? See Long-running agent jobs.
- Who is accountable when one is wrong — the platform, or the team?
The fourth is the one that reveals whether this is a real platform question. A platform that provides capability without providing evaluation and guardrails has moved the risk without reducing it.
The shape
team's agent definition
│
┌────────▼────────┐
│ control plane │ registry, versions
└────────┬────────┘
│
┌────────▼────────┐
│ agent runtime │ loop, tenant caps
└───┬─────────┬───┘
│ │
┌────▼───┐ ┌───▼───────┐
│gateway │ │tool broker│
└────────┘ └───────────┘
models actions
observability + evals span bothThe split that matters is control plane versus data plane. Agent definitions, prompts, tool grants and model choices are configuration, changed rarely and reviewed. Runs are traffic. Conflating them is why teams end up deploying code to change a prompt.
The five things the platform owns
1. The gateway
One place for provider keys, routing, retries, caching and per-tenant budgets. Without it, cost attribution is impossible and every team re-implements fallback. See The LLM gateway and caching.
2. Tool authorisation
The security core. Tools are declared centrally with a schema and an owner, and a tool call is authorised against the end user, never against what the model asked for. The model proposes; the platform decides.
This is what makes prompt injection survivable: an injected instruction can make the agent request a refund, and the authorisation layer still refuses because that user cannot refund that order. See Prompt injection.
3. Evaluation as a platform service
Each team brings a golden set; the platform runs it in CI, stores results per version, and blocks a promote on regression. Centralising the harness while teams own the cases is the split that works — see The eval and observability tooling.
4. Observability with the AI-specific signals
Traces carrying tenant, feature, prompt version, model version, retrieved chunks, tool calls and cost. These are what make an incident investigable, and they must be attached at call time — you cannot add them later.
5. Isolation and fairness
Per-tenant concurrency and token budgets, because the failure mode is one tenant’s runaway loop consuming the shared provider quota and taking everyone down. See Concurrency and backpressure.
The arithmetic
Worth doing out loud, because it decides the architecture:
100 tenants x 500 runs/day = 50k runs/day
x 12 model calls per run = 600k calls/day
≈ 7/sec average
≈ 35/sec at peakSeven calls a second is nothing for a web service and a lot against a provider’s tokens-per-minute quota. The bottleneck is the provider limit, not your compute — which means the design centres on queuing, per-tenant fairness and routing cheap work to small models, not on horizontal scaling.
A run that averages 12 calls at 30k tokens is 360k tokens; at 50k runs a day that is an eight-figure monthly token count, and the cost conversation starts before the reliability one.
Rollout, and the thing that is different
Agents are non-deterministic, so the usual staged rollout needs an extra step:
| Stage | Gate |
|---|---|
| Eval suite | no regression on the golden set |
| Shadow | run alongside, compare, no user impact |
| Canary | small traffic slice, watch quality signals |
| Full | with a kill switch |
Shadow mode is the one people skip and it is the most valuable, because it is the only stage that compares old and new behaviour on real traffic without a user seeing the result.
The kill switch is not optional: the platform must be able to disable one tenant’s agent, or the AI path entirely, without a deploy — see Incident response for AI systems.
Failure modes to name
- A tenant’s runaway loop eats the shared quota. Per-tenant caps.
- Prompt injection via ingested data reaches a privileged tool. Authorise against the user.
- A provider degrades and every agent slows together. Fallback at the gateway, and a cheaper model tier.
- Silent quality regression from a prompt change. Eval gates and shadow.
- Cost surprise discovered at month end. Attribution and pre-call budgets — see Cost attribution.
The closing position
“The platform’s value is not the agent loop — that is a library. It is the gateway, the tool authorisation boundary, the eval harness and the traces. Those are what let a regulated business run agents it can explain, cap and switch off.”
That sentence is the design answer. Anyone can wire an agent loop; the platform question is about the things around it.
Related
Interview angle 5
- “Design a platform for running agents.” - split control plane from data plane, then name the five things the platform owns: the model gateway, tool authorisation, evaluation as a service, tracing with AI-specific fields, and per-tenant isolation. The agent loop itself is a library, not the product.
- “Where does security live?” - in tool authorisation, against the end user rather than against what the model requested. That is what makes prompt injection survivable: the injected instruction still produces a request the authorisation layer refuses.
- “What’s the capacity bottleneck?” - the provider’s tokens-per-minute quota, not your compute. A hundred tenants at 500 runs a day is only about 35 calls a second at peak, which is trivial to serve and significant against a provider limit — so the design is about queuing and fairness, not horizontal scaling.
- “How do teams ship a change safely?” - eval suite gate, then shadow mode comparing old and new on real traffic with no user impact, then a canary, then full with a kill switch. Shadow is the step people skip and the only one that compares behaviours on real input.
- “What would you build first?” - the gateway and the traces, because without cost attribution and investigable traces you cannot run the thing, and both have to be attached at call time rather than reconstructed later.