Which model for which task
“What models do you use for what?” is a cost-and-judgement question wearing a trivia costume. Naming one frontier model for everything says you have never had to defend an inference bill.
The 2026 production norm is routing: classify the request, send the easy majority to a small model, escalate only what needs it.
Match the model to the job
| Task | Reach for |
|---|---|
| Classify, extract, route | small, fast model |
| Summarisation, rewriting | small to mid |
| Reasoning, code, planning | frontier |
| Agentic tool use | frontier, or a strong mid |
| Embedding | a dedicated embedding model |
| Reranking | a cross-encoder, not an LLM |
Two of those rows are the ones people get wrong.
Embeddings are not a chat model’s job. A dedicated embedding model is orders of magnitude cheaper and better at the task. Asking an LLM to judge similarity is the expensive, worse version.
Reranking wants a cross-encoder, which scores the query and document together in one pass. Using a frontier LLM to reorder ten candidates works and costs about a hundred times more than the model built for it.
Instruction-following is what scales with size
Bigger models are not uniformly “smarter” in ways you need. What reliably improves with capability is following multi-step instructions and choosing tools correctly. A small model extracts a field from an invoice fine; give it a six-tool agent loop and it will call the wrong one.
So the practical test is not “is this task hard” but “how many decisions am I asking the model to chain”.
Routing, concretely
async def choose(req: Request) -> Model:
if req.needs_reasoning or req.tool_count > 3:
return models["frontier"]
if req.kind in ("classify", "extract", "route"):
return models["small"]
return models["default"]Classify by task type first, because it is free. A cheap-classifier or length-based route is a refinement, not the starting point.
Measure the escalation rate. If 90% of requests reach the frontier model, the router is costing you latency and buying nothing — either the routing rule is wrong or the workload genuinely is hard, and both are useful to know.
That means the router emits a metric, not just a model:
model = await choose(req)
ROUTED.labels(
tier=model.tier, kind=req.kind
).inc()Then one query answers “is the router earning its keep”:
sum(rate(routed_total{tier="frontier"}[1h]))
/ sum(rate(routed_total[1h]))Alert when it climbs. A router nobody measures degrades into an expensive
if False.
Gotcha: route on the task, not on the user or the tier. Routing paying customers to a better model sounds reasonable and produces support tickets about inconsistent answers that nobody can reproduce.
Fallbacks are a different mechanism
Routing chooses by task; fallback reacts to failure. A provider 429 or a timeout should fail over to a comparable model, not silently downgrade a reasoning request to a small one. Keep the two decisions separate in code, or a provider outage quietly changes your output quality with no signal.
See Model-provider abstraction and LLM resilience.
What actually moves the bill
In rough order of effect:
- Route. Most requests do not need the biggest model.
- Cap output. Output tokens cost several times input, so response format is a recurring charge — verbose JSON keys are real money.
- Cache the prompt prefix. A long stable system prompt or retrieved context is cached at a large discount by most providers.
- Shorten the context. Fewer, better-retrieved chunks beat top-50, and they improve answers too.
- Batch anything not interactive; batch APIs are heavily discounted.
Only then consider a weaker model. Quality regressions cost more to diagnose than the tokens saved.
Saying it well
State the trade-off, not the brand:
“We route by task. Extraction and classification go to a small model, anything multi-step goes to a frontier one, and we track the escalation rate. Embeddings and reranking use models built for those jobs rather than a chat model. That kept p95 latency down and the majority of requests off the expensive path.”
Note: as of 2026-08, prefer naming a model class — small, mid, frontier — over a specific version when the point is capability. Versions move faster than the reasoning does; see Stack baseline — 2026-2027 for the current table.
Related
Interview angle 5
- “Which models do you use for which tasks?” - route by task. Small and fast for classification, extraction and routing; frontier for multi-step reasoning, code and agentic tool use; a dedicated embedding model for vectors; a cross-encoder for reranking. Naming one frontier model for everything is the answer that says you have never owned the bill.
- “What actually gets better as the model gets bigger?” - following multi-step instructions and choosing tools correctly. So the question is not how hard the task is but how many decisions you are chaining — a small model extracts a field reliably and falls apart in a six-tool agent loop.
- “How do you decide when to escalate?” - classify by task type first because it costs nothing, then refine with a cheap classifier if needed. Then measure the escalation rate: if almost everything escalates, the router is adding latency and saving nothing.
- “Routing or fallback?” - different mechanisms. Routing picks by task; fallback reacts to a 429 or timeout. Conflate them and a provider outage silently downgrades a reasoning request to a small model with no signal that quality dropped.
- “How would you cut inference cost?” - route first, then cap output tokens (output costs several times input), then cache the stable prompt prefix, then shorten retrieved context, then batch anything non-interactive. Swapping in a weaker model comes last, because a quality regression costs more to diagnose than the tokens saved.