AI & ML / MLOps & LLMOps / 03_llm_observability.md

LLM observability

Updated 7 interview angles 8 min read source
On this page8
  1. OpenTelemetry GenAI conventions
  2. The mechanism: how the trace stays connected
  3. What to capture
  4. Where this breaks in production
  5. The tooling
  6. Dashboards worth having
  7. Related
  8. Interview angle

LLM observability

Article.

Verified 2026-08

Verified 2026-08. This area moves; re-check before quoting.

An LLM call produces a span with a duration and a status, like any HTTP call. That tells you nothing you need: not which prompt version, not which model, not the token count, not the tool-call chain, not what it cost.

For an agent it is worse. One user request becomes a tree of model calls, tool executions, retrievals and possibly sub-agents, and without a trace you cannot answer the only question that matters when it misbehaves — why did it do that?

OpenTelemetry GenAI conventions

The vendor-neutral standard: span names, attribute keys, metric instruments and event names for model invocations, tool executions, agent runs, retrieval and memory operations.

python 
with tracer.start_as_current_span("chat gpt-model") as span:
    span.set_attribute("gen_ai.system", "openai")
    span.set_attribute("gen_ai.request.model", MODEL_ID)
    span.set_attribute("gen_ai.request.temperature", 0.0)

    response = client.chat.completions.create(...)

    usage = response.usage
    span.set_attribute("gen_ai.usage.input_tokens", usage.prompt_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", usage.completion_tokens)
    span.set_attribute("gen_ai.response.finish_reasons", ["stop"])

The status, precisely. As of 2026 the GenAI and MCP semantic conventions are still experimental, with no public stabilisation date. In v1.42.0 (12 June 2026) all gen_ai.* attributes moved out of the main semantic-conventions repository into a dedicated GenAI repo — an organisational change giving fast-moving work its own release cadence, not a graduation to stable.

The consequence: attribute names can still change between releases. Adopt them anyway — the alternative is proprietary lock-in — but pin your instrumentation version and expect to update. “Use the OTel GenAI conventions” without that caveat is the common half-answer.

The mechanism: how the trace stays connected

This is where agent tracing actually breaks, and it is nothing to do with attribute names.

OpenTelemetry tracks the current span in a context variable. A child span finds its parent by reading that variable, which works perfectly inside one coroutine and stops working the moment the work moves.

python 
# Propagates. create_task copies the current context.
task = asyncio.create_task(call_tool(args))

# Propagates. to_thread copies the context into the worker thread.
result = await asyncio.to_thread(blocking_sdk_call, args)

# Does NOT propagate. submit() runs with whatever context the pool thread has.
future = pool.submit(blocking_sdk_call, args)

The failure is not an error. You get a second, orphaned trace with one span in it, and the tool call simply vanishes from the agent’s tree — so the trace you open during an incident is missing exactly the step that failed.

Crossing a process boundary

Handing work to Celery, arq or a queue leaves the process entirely, so the context variable cannot follow. The context has to travel in the message:

python 
# producer
carrier: dict[str, str] = {}
TraceContextTextMapPropagator().inject(carrier)
await queue.enqueue("summarise", doc_id, trace_ctx=carrier)

# consumer
ctx = TraceContextTextMapPropagator().extract(job["trace_ctx"])
with tracer.start_as_current_span("summarise", context=ctx):
    ...

That traceparent string is the whole mechanism — a trace id, a span id and flags. Anything that carries a header or a JSON field can carry it, which is why HTTP propagation works automatically and queue propagation does not: nobody wrote the injection for your message format.

If your agent traces look fragmented, this is why, and it is worth checking before blaming the tooling.

What to capture

Level Attributes
Request trace id, tenant, feature, prompt version
Model call model, tokens, finish reason, latency, TTFT, cost
Tool call name, arguments, result size, latency, success
Retrieval query, top-k, scores, chunk ids
Agent step number, total steps, termination reason
Outcome validation pass, groundedness, escalation

Retrieved chunk ids are the underrated one. When an answer is wrong, knowing exactly which chunks were in the context window is the difference between debugging in minutes and guessing for a day.

Cost per span is the other. It lets you attribute spend to a feature, a tenant or a user, which is how you find the one workflow burning the budget — and cost is not a field any provider hands you, so compute it from tokens at span end.

Where this breaks in production

Streaming makes span duration lie

With a streamed response, the span ends when the stream closes. That duration is not what the user experienced — they saw the first token far earlier and started reading.

Record both, explicitly:

python 
start = time.perf_counter()
first_token_at: float | None = None

async for chunk in stream:
    if first_token_at is None:
        first_token_at = time.perf_counter()
        span.set_attribute("gen_ai.server.time_to_first_token", first_token_at - start)
    yield chunk

span.set_attribute("gen_ai.server.request_duration", time.perf_counter() - start)

TTFT drives perceived speed; total duration drives cost and capacity. A dashboard with only one of them will tell you the system is fine while users say it is slow, or the reverse.

The related trap: if the client disconnects mid-stream, the span may never end and never export. Close it in a finally.

Sampling throws away the traces you need

Agent traces are large — dozens of spans, some with prompt-sized attributes. At volume you cannot keep all of them, and the obvious fix makes things worse.

Head-based sampling decides at the root, before anything has happened, so a 1% rate keeps 1% of traces chosen at random — and discards 99% of your errors. You are left sampling exactly the boring successful requests.

Tail-based sampling decides after the trace completes, so you can keep every trace that errored, exceeded a latency threshold, hit the step limit, or failed validation, and sample the rest at 1%. It costs a collector that buffers whole traces in memory, and that is the right trade.

Traces are a PII store

Full prompts and responses are simultaneously the most useful debugging data and the largest privacy liability you will create.

python 
if should_sample(trace_id, rate=0.01) and not contains_pii(prompt):
    span.set_attribute("gen_ai.prompt", redact(prompt))

Metadata always, content on a small redacted sample with short retention. Two consequences teams discover late: an erasure request under GDPR covers your trace backend as well as your database, and most have never tried deleting a user’s data from it. And span attributes have size limits — a long prompt is silently truncated, so what you kept may not be what you needed.

See PII, privacy and the EU AI Act.

The tooling

Tool Character
LangSmith deep LangChain/LangGraph tie-in, eval built in
Langfuse open source, self-hostable, OTel-compatible
Logfire Pydantic’s OTel platform; app and LLM in one view
Phoenix open source, strong on eval and drift
Braintrust evaluation-centric
Your APM + OTel LLM spans beside the rest of the system

Depth on instrumentation, self-hosting and cost is in Langfuse, LangSmith and Logfire.

Both is the normal answer. OTel keeps the model call in the same trace as the database and queue spans around it, which is what you need at 3am. A specialised tool gives you prompt playgrounds, conversation-tree UI and evaluation integration, which is what you need on a Tuesday afternoon.

The trade not taken

You could skip tracing and log structured events instead — cheaper, simpler, and adequate for a single model call behind an endpoint.

It stops being adequate the moment there is a tree. Logs give you records with no parent-child relationship, so reconstructing “this retrieval fed that model call which called that tool” means joining on ids you remembered to log consistently. That is a trace, rebuilt by hand, badly. The threshold is whether one request produces more than one model call — below it, logs; above it, spans.

Dashboards worth having

  • Cost — spend by feature, tenant and model; tokens per request over time.
  • Latency — p50/p95 end to end, and TTFT separately.
  • Quality — schema-validation pass rate, refusal rate, sampled groundedness.
  • Agent health — mean steps per run, step-limit exhaustion, tool error rate.

Step-limit exhaustion is the best leading indicator you get. A rising rate means agents are increasingly failing to finish, and it moves before task success falls and well before users complain.

Interview angle 7

  • “How do you trace an LLM application?” — OpenTelemetry with the GenAI semantic conventions: model, tokens, cost, latency and finish reason per call, plus tool calls, retrieval chunk ids and agent steps, in the same trace as the HTTP and database spans around them.
    • Follow-up: “Are those conventions stable?” — no. They are still experimental with no stabilisation date, and the v1.42.0 move to a dedicated repo was about release cadence, not graduation. Adopt them, pin the instrumentation version, expect attribute renames.
  • “Your agent trace only shows the first two steps. What happened?” — context propagation. OTel carries the current span in a context variable; asyncio.create_task and to_thread copy it, a raw ThreadPoolExecutor.submit does not, and nothing crosses a process boundary on its own.
    • Follow-up: “So how do you trace work handed to Celery?” — inject the traceparent into the message and extract it in the worker. HTTP does this automatically; your queue does it only if you wrote it.
  • “You stream responses. What does the span duration measure?” — time until the stream closed, which is not what the user felt. Record time-to-first-token as its own attribute: TTFT drives perceived speed, total duration drives cost and capacity, and a dashboard with one of them misleads you.
  • “You cannot afford to keep every trace. What do you sample?” — tail-based, not head-based. Deciding at the root keeps a random 1% and throws away 99% of your errors. Deciding after completion lets you keep everything that errored, was slow, or hit the step limit, and sample the rest.
  • “Do you log prompts and responses?” — metadata always, content on a small redacted sample with short retention. Traces holding prompts are a PII store: an erasure request reaches your trace backend, and long prompts hit span attribute size limits and truncate silently.
  • “Which metric warns you an agent is degrading before users complain?” — step-limit exhaustion rate, with mean steps per run trending up beside it. Both move before task success falls.
  • “When is tracing overkill?” — one model call behind an endpoint; structured logs are enough. The threshold is whether a single request produces more than one model call, because logs cannot express a parent-child tree without you rebuilding one by hand.
Previous
Monitoring and drift