Backend / Observability / 14_cost_and_finops.md

Cost as a signal

Updated 6 interview angles 5 min read source
On this page7
  1. The three stages
  2. Inform: attribution or nothing
  3. Optimize: waste first, discounts second
  4. The LLM line item
  5. Operate: where it usually breaks
  6. Related
  7. Interview angle

Cost as a signal

Verified 2026-08

Verified 2026-08.

Latency has an owner, a dashboard and an alert. Cost usually has an invoice, once a month, addressed to someone who cannot change it.

FinOps is the practice of treating cost as an engineering signal — measured continuously, attributed to a team, and acted on by the people who caused it. The interview version is not “how do you save money”; it is how do you make cost visible enough that saving it becomes an ordinary engineering decision.

The three stages

text 
INFORM     what did we spend, and who spent it
   ↓
OPTIMIZE   remove waste, then buy the rest cheaper
   ↓
OPERATE    make it continuous, not a quarterly panic

Teams that skip straight to Optimize get a one-off saving that erodes within two quarters, because nothing tells them when it comes back. Inform is the stage that makes the other two repeatable.

Inform: attribution or nothing

You cannot act on a number nobody owns. Attribution is the whole first stage, and it is a tagging problem before it is a finance problem.

hcl 
# Tags are the join key between the invoice and the org chart.
default_tags {
  tags = {
    team        = "payments"
    service     = "checkout-api"
    environment = "prod"
    cost_center = "CC-4417"
  }
}

Enforce them at provisioning time — in the Terraform module or via policy — and untagged resources become an exception rather than the default. Retro-tagging a year of infrastructure is a project; tagging at creation is a line of config.

The metric that turns cost into an engineering conversation is unit economics: cost per request, per tenant, per model inference. Total spend going up is ambiguous — it might be growth. Cost per unit going up is a regression, and it can be alerted on like any other.

Optimize: waste first, discounts second

The order matters and is frequently reversed.

Lever Typical return Effort
Delete idle resources large low
Rightsize over-provisioned large low
Storage lifecycle policies medium low
Autoscaling downward medium medium
Commitment discounts large commits you
Architecture change largest high

Buying a three-year commitment for capacity you should have deleted locks in the waste. Clean up first, measure the new steady state, then commit to that.

The unglamorous wins are the reliable ones: non-production environments running at nights and weekends, orphaned volumes and load balancers, snapshots with no lifecycle rule, and logs retained at full fidelity for a year because nobody picked a number.

The LLM line item

For an AI product this is often the fastest-growing cost in the business, and it behaves differently from infrastructure: it scales with usage rather than with provisioned capacity, so there is nothing idle to delete.

The levers, in order of return:

  1. Route by difficulty. A smaller model for the easy majority is the single biggest lever, and it needs no serving expertise — see Which model for which task.
  2. Cache. Exact, prefix and semantic caching each remove whole requests. Prefix caching also wants the prompt shaped stable-content-first, which is free to do and easy to break.
  3. Shorten the context. Cost is roughly linear in tokens, and conversation history re-sent every turn grows as a triangular number — see Context windows.
  4. Then serving efficiency — batching, FP8 KV cache — which only applies if you host the model yourself.
python 
# Attribute spend the same way you attribute latency: per call, with a key
# that ties back to a team and a feature.
log.info(
    "llm_call",
    feature="ticket_summary",
    team="support-platform",
    model=model,
    input_tokens=usage.input_tokens,
    cached_tokens=usage.cache_read_input_tokens,
    output_tokens=usage.output_tokens,
    usd=estimate_cost(model, usage),
)

Without that record, “the AI bill went up 40%” cannot be traced to a feature, and the only available response is to ask everyone to use it less.

Operate: where it usually breaks

A dashboard nobody owns. Cost visibility with no named owner per service produces a report that is read once. Attribution has to reach a team that can act.

Alerting on total spend. Total spend rises with growth, so the alert either fires constantly or is set so high it never fires. Alert on cost per unit and on anomalies against the trend, not on the absolute number.

Optimising the visible instead of the expensive. Compute is what people look at; egress, cross-AZ traffic, NAT gateways, per-request logging and idle non-production frequently exceed it. Read the bill before optimising from intuition.

Savings that quietly reverse. Someone rightsizes, and six months later the instances are large again because nothing watched. The saving is only real if a metric would show it coming back.

Cost work that costs more than it saves. Two engineer-weeks to save $200/month is a loss. Estimate the return before starting, the same as any other project.

Interview angle 6

  • “What is FinOps?” - treating cost as an engineering signal rather than a monthly invoice: measured continuously, attributed to a team, and acted on by the people who caused it. Three stages - inform, optimize, operate - and skipping inform is why one-off savings erode.
  • “Where do you start on a cloud bill?” - attribution, which is a tagging problem. Enforce tags at provisioning time so untagged resources are the exception; retro-tagging a year of infrastructure is a project. Then unit economics - cost per request or per tenant - because total spend going up might just be growth.
  • “Rightsizing or reserved instances first?” - rightsizing, always. A three-year commitment on capacity you should have deleted locks the waste in for three years. Clean up, measure the new steady state, then commit to that.
  • “How would you alert on cost?” - on cost per unit and on anomalies against trend, never on total spend. Total rises with growth, so an absolute threshold either fires constantly or is set high enough to be useless.
  • “Your LLM bill is growing faster than usage. What do you look at?” - the per-call record first: model, feature, input and output tokens, cached tokens. Then the levers in order - route easy requests to a smaller model, cache, and shorten the context, remembering that re-sending conversation history grows the token count as a triangular number rather than a linear one.
    • Follow-up: “Which of those needs no serving expertise?” - routing. Halving the parameter count halves the bytes read per token, which is the actual bottleneck, and it is a change in application code rather than in the serving stack.
  • “What gets optimised that should not?” - whatever is most visible. Compute is what people look at, while egress, cross-AZ traffic, NAT gateways and idle non-production environments quietly cost more. Read the bill before optimising from intuition - and check the work is worth more than it costs, because two engineer-weeks to save $200 a month is a loss.
Previous
Structured logging in Python