Cut LLM spend without losing quality

The brief: the LLM bill is $40k a month and needs to be $12k. Quality must not measurably drop. You have a working product and real traffic.

The levers themselves are not the hard part — they are enumerated in Which model for which task and the answer is roughly always route, cap output, cache the prefix, shorten context, batch.

What makes this a senior question is the second half of the brief. Anyone can make it cheaper. The job is making it cheaper and being able to prove nothing broke, because a quality regression found by customers costs more than the tokens ever saved.

Do the accounting before touching anything

A 70% target applied uniformly is the wrong shape. Spend is almost never uniform, and the first useful artifact is a breakdown by feature and by call site, not an invoice total.

# Log usage per call, tagged with what caused it. The provider's reported
# usage is authoritative - never your own token count.
log.info(
    "llm_call",
    feature=feature,          # "summarise" | "chat" | "extract"
    model=resp.model,
    input_tokens=resp.usage.input_tokens,
    cached_tokens=resp.usage.cache_read_input_tokens,
    output_tokens=resp.usage.output_tokens,
    trace_id=trace_id,
)

Attribution mechanics, including where an agent’s cost hides, are in Cost attribution.

Expect the breakdown to be lopsided: one feature at 70% of spend, or a retry path nobody costed, or an agent whose median trajectory is four tool calls and whose p99 is forty. Two findings recur often enough to check first:

• Retries are billed. A failed structured-output parse that retries twice triples that call. It shows up as volume, not as errors.
• Context is resent every turn. A chat feature with no prefix caching pays for the whole history on every message, so cost grows quadratically in turns while the user perceives a linear conversation.

The order that preserves quality

Ordered by effect per unit of risk, not by effect alone:

The first row is retries, dead calls and abandoned requests you are still billed for. The first five levers are close to free — they change what you send, not what answers. Routing and distillation change which model answers, and that is where the regression harness has to exist before you ship, not after.

Note the fourth row. Sending fewer, better-retrieved chunks is a cost lever that usually raises quality, because top-50 context dilutes attention and buries the evidence. It is the rare case where the cheap direction is also the correct one.

Distillation and prompt compression

Two levers the enumerated list does not cover, both real and both riskier than they look.

Distillation is fine-tuning a small model on the frontier model’s outputs for one narrow task. It works well for classification, extraction, and fixed format generation — jobs where the output space is constrained and you have traffic to harvest.

It fails for anything open-ended or multi-step, and it inherits every bias of the teacher including its mistakes. Budget for a labelled holdout the teacher never saw, or you are measuring agreement with the teacher rather than correctness.

Prompt compression removes low-information tokens from context before sending it. It is genuinely effective on verbose retrieved documents and genuinely dangerous on anything where a dropped token changes meaning — a negation, a units marker, a row of a table. Treat it as a per-feature experiment, never a global setting, and never apply it to user instructions.

Proving nothing broke

This is the deliverable. Without it you have a cost reduction and a hypothesis.

Freeze a golden set first. Build it before you change anything, from real traffic, weighted to match production distribution rather than uniformly. Construction is covered in Building eval sets.

Score paired, not absolute. Absolute LLM-as-judge scores are noisy enough to hide a real regression. Run the old and new configurations on the same inputs and compare them head to head, with the presentation order randomised — judges have a position bias. Biases and mitigations in LLM as judge.

Set the bar before you start

Decide the threshold in advance. “No measurable quality loss” has to become a number before you see the result, or you will rationalise whatever you get. Write it as a test, so the decision is made once:

def accepts(result: EvalRun) -> bool:
    """The gate, fixed before the experiment runs."""
    # Ties count for the cheap side: we are proving "not worse", not "better".
    win_rate = (result.cheap_wins + result.ties) / result.n_pairs
    return (
        win_rate >= 0.48                      # paired, order randomised
        and result.schema_valid >= baseline.schema_valid
        and result.citation_rate >= baseline.citation_rate
        and result.refusal_rate <= baseline.refusal_rate * 1.05
    )

Note that the judge threshold is the loose one and the deterministic checks are the strict ones. That is the right way round: the judge is noisy, and the mechanical checks are not.

Keep deterministic checks separate. Schema validity, citation presence, refusal rate, and format compliance are cheap, exact, and non-negotiable. A cheaper model that is equally liked by a judge but emits invalid JSON 4% of the time is not equal, and only the deterministic check catches that.

Ship it as a routed experiment. Send 5% of traffic to the cheap path, compare online signals — thumbs, abstention rate, escalation rate, retry rate — and hold there long enough to see them move. The escalation rate is the one that matters for routing: if the small model punts to the large one 40% of the time, you have added latency and saved nothing.

Where teams get this wrong

• Optimising the wrong feature. Cutting 60% off the feature that is 8% of spend is a week for 5%. Do the breakdown first.
• Quality measured only by a judge. Add the deterministic checks; they catch the failure class judges are blind to.
• A single number as the target. “70% cheaper” with no latency or quality bound is satisfiable by making the product worse, and someone will.
• Ignoring latency as a cost. A cheaper batch path that adds 20 seconds to an interactive flow moved cost onto the user.
• No rollback. Model routing behind a config flag, not a deploy, so a regression is a one-minute revert.

The closing position

State it as a sequence with a gate: measure and attribute, take the free levers first, then route with a paired eval and a pre-declared threshold, ship at 5% and watch escalation and abstention before ramping. The sentence that lands is that you can prove the cheap system is not worse, because that is the part that is hard and the part most candidates skip.

Related

• Which model for which task
  • the levers, and routing mechanics.
• Cost attribution
  • per-call attribution and budget enforcement.
• The LLM gateway and caching
  • prompt, exact and semantic caching as distinct layers.
• Building eval sets
  • the golden set and paired comparison.
• Online evaluation and experiments
  • running the 5% experiment and reading the signals.
• Distillation and synthetic data
  • distillation mechanics.