AI & ML / MLOps & LLMOps / 04_cost_attribution.md

Cost attribution

Updated 5 interview angles 4 min read source
On this page7
  1. Attribute at the call, not from the invoice
  2. Use reported usage, not your own count
  3. The unit that matters is per outcome
  4. Budgets are enforcement, not reporting
  5. The three questions to be ready for
  6. Related
  7. Interview angle

Cost attribution

The provider bill is one number per organisation per month. Turning it into “team X spent this, customer Y is unprofitable, that feature costs 4p per call” is a build, and it is usually the first thing a CTO asks for once AI spend becomes visible.

Attribute at the call, not from the invoice

You cannot reconstruct attribution afterwards. The dimensions have to be attached when the request is made:

python 
await llm.complete(
    prompt,
    metadata={
        "tenant_id": ctx.tenant,
        "feature": "doc_summary",
        "user_id": ctx.user,
        "trace_id": ctx.trace_id,
        "env": settings.env,
    },
)

Every response reports usage — prompt tokens, completion tokens, cached tokens. Multiply by the model’s rates, store the row, and the questions become a GROUP BY.

Dimension Answers
Tenant who is expensive, who is unprofitable
Feature which product area to optimise
Model is routing working
Environment how much is staging burning

The environment row catches a specific embarrassment: a load test or a runaway dev loop against the production key, discovered at month end.

Use reported usage, not your own count

Estimating tokens with a tokenizer is a fallback, not a plan. It drifts from what you are billed for — system-added tokens, tool schemas, image tiling and provider-side reformatting all land in the invoice and not in your estimate.

Reconcile monthly against the provider’s dashboard. A drift of a few percent is normal; a drift of thirty percent means you are missing a call path, and it is better to find that yourself.

Cached tokens are billed at a different rate, so a cost model that ignores the cache overstates spend and hides the benefit of prompt ordering. See The LLM gateway and caching.

The unit that matters is per outcome

Cost per token is an implementation detail. The business question is cost per unit of value:

text 
cost per resolved ticket
cost per document processed
cost per active user per month

That framing is what makes the number actionable. “We spend £12k a month on LLMs” invites a cut; “each resolved ticket costs 30p against £4 for a human” invites investment. Same data, different conversation, and the second is the senior one.

It also exposes the failures worth finding: a feature where cost per outcome exceeds its price, or a tenant whose usage pattern makes their plan lose money.

Where an agent’s cost hides

Per-call cost misleads for agents, because one logical task is many calls. Aggregate to the run, and include the retries — a run that takes ten steps and retries twice is thirty calls against one unit of work. See Concurrency and backpressure.

Budgets are enforcement, not reporting

A dashboard tells you about the money after it is spent. The control is a budget checked before the call:

python 
if await budget.remaining(tenant) <= 0:
    raise BudgetExceeded(tenant)

Per tenant, per feature, and globally. A gateway is the natural place, because it already sees every request.

Gotcha: decide what happens at the limit before you hit it. Hard-fail, degrade to a cheaper model, or queue until the window resets are all defensible; discovering you never decided, during an incident, is not.

Alerting on the rate of spend catches a runaway loop within minutes, which a daily total does not.

The three questions to be ready for

  1. “What does this feature cost per use?” — needs feature tagging.
  2. “Which customers are unprofitable?” — needs tenant tagging joined to your billing data.
  3. “Where did last month’s increase come from?” — needs history, so keep the per-call rows long enough to compare periods.

If your instrumentation cannot answer all three, it is not attribution, it is a total.

Interview angle 5

  • “How would you track AI spend per customer?” - attach tenant, feature and environment to every call and store the reported usage per request. You cannot reconstruct it from the invoice afterwards, so the dimensions have to be attached at call time.
  • “Would you count tokens yourself?” - only as a fallback. Use the provider’s reported usage, because system-added tokens, tool schemas and image tiling land in the bill and not in your estimate. Reconcile monthly against the dashboard; a thirty percent drift means a call path you are not instrumenting.
  • “What’s the metric that matters?” - cost per outcome, not per token. “Each resolved ticket costs 30p against £4 for a human” is a different conversation from “we spend £12k a month”, and it is the one that gets a feature funded.
  • “Why is per-call cost misleading for agents?” - one logical task is many calls, and retries multiply it. Aggregate to the run, including retries, or a ten-step run that retried twice looks like thirty cheap calls rather than one expensive unit of work.
  • “Reporting or enforcement?” - both, but the control is a budget checked before the call, per tenant and globally, usually at the gateway. Decide in advance whether hitting it hard-fails, degrades to a cheaper model or queues — deciding during an incident is the failure.
Previous
LLM observability