Worked Design — LLM usage metering

Article.

“Design a system that records LLM usage (tokens, model, latency) per customer, shows it on dashboards, enforces budgets, and feeds billing.” It looks like logging. It is a correctness problem: tokens are money, the provider bills you whether or not you recorded the call, a streamed answer can be cut off before its usage arrives, and cached tokens are priced at a fraction of normal input.

The interviewer listens for where you need ACID (budgets, billing), where eventual consistency is fine (dashboards), how you avoid double counting (idempotency), and how you know your numbers match the provider’s invoice.

Prices below are Anthropic’s published list prices as of October 2026; the design is provider-neutral. The method this follows is in System Design Interview — The Framework.

1. Requirements

Functional

• Every LLM call made by any internal service is recorded: tenant, user, feature, model requested and model served, token counts by type, cost, latency, outcome.
• Dashboards: spend and tokens per tenant, model and feature; latency per model.
• Budgets: a tenant with a monthly cap is stopped (hard) or warned (soft).
• Billing: a monthly invoice line per tenant that finance trusts.

Non-functional

• No lost usage: a missed record is unbilled spend.
• No double counting: retries and redeliveries must not bill twice.
• Metering adds almost nothing to latency: under 5 ms on the request path; the model call itself takes seconds.
• Reconciles with the provider’s own usage report within 1%.
• Prompts and outputs are not stored by default (privacy, size).

2. Estimate

1 M LLM calls/day        ≈ 12/s average, ~35/s peak
event ~500 B (ids, counts, cost, timings)
  → 0.5 GB/day, ~180 GB/year raw (before compression)
per call: 1,000 uncached + 2,000 cached input, 400 output
  on Claude Opus 5.5 ($4 in, $0.20 cache read, $20 out per MTok)
  = 0.0040 + 0.0004 + 0.0080 = $0.0124
  x 1 M/day = $12,400/day ≈ $372,000/month

Two conclusions from the numbers. The write rate is trivial for one Postgres, so the hard parts are correctness and attribution, not throughput. And 1% of error is $3,700 a month, which is why reconciliation is a requirement, not a nice-to-have.

3. Where usage is captured

Every service calls models through one internal gateway (a service, or a shared client library), never the provider SDK directly. That is the only place that sees every call, so it is where usage is recorded, budgets are checked and keys are held. LiteLLM, Portkey and cloud AI gateways are off-the-shelf versions; an in-house one is a few hundred lines.

service ─▶ LLM gateway ─────────────▶ provider
             │ budget check (sync)          │
             │ usage event (async)          │ usage report
             ▼                              ▼
           queue ─▶ ingester ─▶ events ◀─ reconciler
                                  │
                                  └─▶ rollups ─▶ dashboards,
                                                 billing

What the provider tells you, and the traps in it

The Anthropic Messages API returns a usage object with input_tokens, output_tokens, cache_creation_input_tokens and cache_read_input_tokens. Three details decide whether the numbers are right:

• input_tokens excludes cached tokens. Total input is input_tokens + cache_creation_input_tokens + cache_read_input_tokens. Store all three; summing only input_tokens under-reports volume, and pricing all input at the base rate over-reports cost.
• They are priced differently. A 5-minute cache write costs 1.25x the base input price, a 1-hour write 2x, a cache read 0.1x (0.05x on Opus 5.5 and Sonnet 5.5). Batch requests are half price; US-only inference (inference_geo: "us") is 1.1x.
• In a stream, usage arrives in pieces. message_start carries the input counts; message_delta events carry output_tokens, which are cumulative, so the last one wins, not the sum.

Other providers name things differently (OpenAI’s prompt_tokens include cached tokens, reported separately as a subset; reasoning tokens are a subset of output). Normalise at the gateway into one schema, and keep the raw usage object as JSON for audits.

Capturing a stream, including one that is cut off

async def stream_and_meter(client, rec: UsageRecord, **params):
    t0 = time.monotonic()
    try:
        async with client.messages.stream(**params) as stream:
            async for event in stream:
                if event.type == "message_start":
                    u = event.message.usage
                    rec.message_id = event.message.id
                    rec.model_served = event.message.model
                    rec.input = u.input_tokens
                    rec.cache_write = u.cache_creation_input_tokens or 0
                    rec.cache_read = u.cache_read_input_tokens or 0
                elif event.type == "content_block_delta" and rec.ttft is None:
                    rec.ttft = time.monotonic() - t0
                elif event.type == "message_delta":
                    rec.output = event.usage.output_tokens   # cumulative
                yield event
        rec.complete = True
    finally:                                   # runs on cancel and error too
        rec.duration = time.monotonic() - t0
        emit_usage(rec)                        # never blocks the request

If the user disconnects mid-answer, the request is cancelled, but the provider has already generated (and billed) some output. The finally block still emits the record, with complete = False and the last cumulative output count seen, which is an under-estimate; the reconciler corrects the total later. If refusal fallbacks or a router can serve a different model than requested, bill on model_served.

4. Data model

CREATE TABLE usage_events (
  event_id        uuid        NOT NULL,   -- gateway request id + attempt
  occurred_at     timestamptz NOT NULL,
  tenant_id       bigint      NOT NULL,
  feature         text        NOT NULL,   -- "support-bot", "summarise"
  provider        text        NOT NULL,
  model_requested text        NOT NULL,
  model_served    text        NOT NULL,
  provider_msg_id text,
  input_tokens    int NOT NULL, cache_write_tokens int NOT NULL,
  cache_read_tokens int NOT NULL, output_tokens int NOT NULL,
  cost_micros     bigint      NOT NULL,   -- USD x 1,000,000, never float
  price_version   int         NOT NULL,
  ttft_ms int, duration_ms int NOT NULL,
  status          text        NOT NULL,   -- ok | error | cancelled
  complete        boolean     NOT NULL,
  PRIMARY KEY (event_id, occurred_at)
) PARTITION BY RANGE (occurred_at);       -- one partition per month

• Money as integers (micro-dollars). A float sum of a million 0.0124s drifts; finance will find it.
• price_version points to a price table with effective dates. Prices change and models are added monthly; storing the raw token counts plus the version used means any month can be re-priced, and an invoice can be explained line by line.
• Monthly partitions keep the hot month small and let old months move to cheaper storage or be dropped after the retention period.
• At much larger volumes, the events go to a columnar store (ClickHouse, BigQuery) and Postgres keeps only budgets and invoices; at 0.5 GB/day, Postgres with rollups is enough.

Rollups for dashboards

Dashboards never scan raw events. A job (or a continuous aggregate) keeps hourly totals per (tenant, feature, model_served); a month of a tenant’s dashboard is then a few hundred rows. Late events (retries, the reconciler) update past hours, so rollups are recomputed for a trailing window, not only the current hour.

5. Idempotency: count each call exactly once

Two kinds of duplicates, handled differently:

• The same event delivered twice (queue redelivery, ingester crash after insert, gateway retrying the publish). The gateway assigns event_id before the call; the ingester inserts with ON CONFLICT (event_id, occurred_at) DO NOTHING. Delivery is at least once; storage is exactly once.
• The same user request sent to the provider twice (a timeout, then a retry). Those are two real, separately billed calls: record both, as attempts of one request (event_id = request_id + attempt), so retry cost is visible per feature. Collapsing them hides real spend.

The usage event must not be lost between the gateway and the queue. Either the gateway publishes to a durable queue with acknowledgement before it returns, or it writes to a local spool that a sidecar ships; a crash with an in-memory buffer loses usage the provider will still bill.

6. Budgets: the part that needs ACID

A hard cap (“stop at $500 this month”) must hold under concurrency: twenty parallel calls each seeing $499 spent would all pass a read-then-check. Reserve before the call, settle after:

-- before the call: reserve the worst case (input estimate + max_tokens output)
UPDATE budgets
   SET reserved_micros = reserved_micros + :worst_case
 WHERE tenant_id = :t AND month = :m
   AND spent_micros + reserved_micros + :worst_case <= limit_micros
RETURNING tenant_id;                  -- no row → over budget, reject 429

-- after the call: release the reservation, add the actual cost
UPDATE budgets
   SET reserved_micros = reserved_micros - :worst_case,
       spent_micros    = spent_micros + :actual
 WHERE tenant_id = :t AND month = :m;

• The single conditional UPDATE is atomic, so concurrent calls cannot overspend by more than the reservations allow.
• Reserving max_tokens worth of output is pessimistic; a tenant near the cap is refused slightly early, which is the safe direction.
• A crashed gateway leaves a reservation behind; a sweeper releases reservations older than the longest possible call.
• If this row becomes hot (one tenant at hundreds of calls per second), move the counter to Redis with the same check in a Lua script, and persist the settled totals to Postgres.
• Soft limits (alert at 80%) need none of this: they read the rollups and tolerate minutes of lag.

7. Reconciliation: proving the numbers

The provider’s own report is the ground truth for what you pay. Anthropic’s Usage and Cost Admin API returns tokens by model, workspace, API key and service tier in 1-minute, 1-hour or 1-day buckets, with data usually available within about 5 minutes; the cost report is daily, in USD. An hourly job:

1. Sums your usage_events per hour, model and API key.
2. Fetches the provider’s usage for the same hour, with a lag of an hour or two so late data has arrived.
3. Alerts if any bucket differs by more than 1%, and records the difference.

Typical causes of a gap: calls that bypassed the gateway (a team using its own key, found by grouping the report by API key), cancelled streams whose output was under-counted, a price table missing a new model, or cached tokens counted as normal input. One API key per environment and service makes the report attributable.

8. Latency and observability

Per call, record time to first token (what a user feels in a chat UI), total duration, and output tokens per second (which separates a slow model from a long answer). Two places, two cardinalities:

• Prometheus histograms labelled by model, feature and status, for alerting and p99s. Not by tenant: thousands of tenants times buckets is a cardinality problem (Prometheus).
• The events table for per-tenant latency and cost questions.
• Traces: the gateway’s span carries OpenTelemetry’s GenAI attributes (gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens). Those conventions are still marked Development, so pin the version you emit.

9. Caching, three different meanings

• Provider prompt caching changes the bill, not the record count: the same prompt’s input is split across cache writes and reads. Track the cache-read share per feature; a drop means a prompt change broke the cached prefix, and cost jumps with no change in traffic.
• Your own response cache (exact or semantic): a hit costs no provider tokens. Record it as an event with cost_micros = 0 and status = 'cache_hit', so request counts, quotas and cache savings stay visible.
• Caching the metering itself: budget state for the hot path in Redis (with the reservation check there), dashboard queries over rollups cached for a minute. Never cache the billing totals.

10. Bottlenecks, failure walk and trade-offs

• Fail open or closed on budgets is a business decision to state explicitly: failing closed stops all customers when Redis blips; failing open risks overspend for minutes. Common answer: fail open up to a small per-tenant allowance kept in gateway memory.
• At 10x (12 M calls/day), raw events move to a columnar store and the ingester batches inserts; budgets and invoices stay in Postgres.
• The trade not taken: provider dashboards only. The provider’s usage page knows tokens per API key, not per tenant or feature; you cannot bill a customer or find the feature that doubled spend from it. It remains the reconciliation source, not the system of record.

Related

• System Design Interview — The Framework - the method this design follows
• Worked Design — Idempotent Payment Flow - idempotency keys and reconciliation for money
• Worked Design — Distributed Rate Limiter - token buckets for request-rate limits
• The transactional outbox - getting events out reliably
• The Three Pillars of Observability - metrics, traces and their cardinality