AI & ML / MLOps & LLMOps / 05_ai_incident_response.md

Incident response for AI systems

Updated 6 interview angles 5 min read source
On this page9
  1. The failure modes that page nobody
  2. Triage: check retrieval first
  3. Mitigations that exist because the model is the dependency
  4. Reproduce from the trace, not the report
  5. Close the loop into evals
  6. The postmortem questions worth asking
  7. Severity needs its own scale
  8. Related
  9. Interview angle

Incident response for AI systems

Standard incident response assumes a binary: the service is up or down. AI systems fail while returning 200s at normal latency, which breaks the detection half of your process and most of the runbook.

The failure modes that page nobody

Failure What monitoring sees
Quality regression nothing
Retrieval broke nothing, answers just get vague
Prompt injection succeeded a successful request
Silent provider update slightly different outputs
Cost spike the invoice, next month

Every row returns 200. This is the argument for the AI-specific signals — see LLM observability — and it is why user feedback is a monitoring channel here rather than a product nicety. A thumbs down wired to a trace id is often the fastest detector you have.

Triage: check retrieval first

The single most useful runbook line for a RAG system, because it splits the problem in half immediately:

text 
bad answer reported
  └─▶ open the trace
      └─▶ right chunk retrieved?
          no  ──▶ chunking / embedding / query
          yes ──▶ prompt / model / ordering

Teams that skip this rewrite the prompt for a week to fix a retrieval bug. Asking “was the evidence even there?” first is the habit worth having.

Mitigations that exist because the model is the dependency

Ordinary rollback does not fully apply — the model is a third-party dependency that can change under you.

  • Pin model versions. A floating alias means the provider can upgrade you mid-incident, or cause one. Pinning makes “what changed?” answerable.
  • Roll back the prompt. Prompts are deployable artefacts and need version history and a rollback path exactly like code.
  • Fall back a tier. Route to a previous model or a simpler non-agentic path.
  • Disable the feature. A flag that turns off the AI path and reverts to the deterministic one is the most valuable mitigation in the list, and the one most often missing.

That last point is the design decision: what does the product do when the AI is switched off? A system with no answer to that has no mitigation short of an outage.

Which means the fallback is a code path you keep working, not a raise:

python 
async def summarise(ticket: Ticket) -> str:
    if not flags.enabled("ai_summary"):
        # degraded, not broken
        return ticket.subject
    try:
        return await llm_summary(ticket)
    except ProviderError:
        return ticket.subject

Test that branch. A fallback nobody exercises is discovered to be broken during the incident it exists for.

Reproduce from the trace, not the report

“It gave a bad answer yesterday” is not reproducible without the artefacts: input, retrieved chunks, prompt version, model version, parameters, output. With them you can replay the exact call.

The corollary for retention: keep enough trace history to investigate a report that arrives a week late, and reconcile that against the PII retention limits in PII, privacy and the EU AI Act. Those two requirements pull in opposite directions and the tension is worth naming.

Gotcha: non-determinism means replaying may not reproduce it. That is not a dead end — sample the same call several times and measure how often the failure occurs. An intermittent 5% failure is a different problem from a deterministic one, and needs a different fix.

Close the loop into evals

The step that distinguishes a team that improves from one that firefights: every incident becomes a test case. The bad input, the expected behaviour, and a regression check in the eval suite.

Without it you fix the same class of problem repeatedly, because a prompt change that fixes today’s issue reopens last month’s and nothing catches it. See Building eval sets.

The postmortem questions worth asking

Beyond the usual timeline and root cause:

  1. How did we find out? If the answer is “a customer told us”, the detection gap is the real finding.
  2. Would the eval suite have caught it? If not, it does now.
  3. Was the model, the prompt, the retrieval or the data at fault? Naming the layer prevents the reflex of blaming “the AI”.
  4. What did it cost? Both in spend and in wrong answers served.

Severity needs its own scale

A wrong answer is not automatically low severity, and an outage is not automatically high. In a regulated domain, confidently wrong is worse than unavailable — an unavailable service is an inconvenience, a fabricated figure in a financial summary is a liability.

Writing that into the severity definitions, rather than inheriting a scale built for uptime, is what makes the process fit the system.

Interview angle 6

  • “How is an AI incident different?” - the system fails while returning 200s at normal latency, so uptime monitoring detects nothing. Quality regressions, broken retrieval and successful injections all look like healthy traffic, which makes user feedback wired to a trace id a monitoring channel rather than a product feature.
  • “Where do you start triaging a bad answer?” - check whether the right chunk was retrieved. That splits the problem in half immediately, and teams that skip it spend a week rewriting prompts to fix a retrieval bug.
  • “What are your mitigations?” - pin model versions so the provider cannot upgrade you mid-incident, roll back the prompt as a versioned artefact, fall back to a simpler path, and have a flag that disables the AI feature entirely. That last one requires deciding what the product does without it, which many teams never do.
  • “What if it doesn’t reproduce?” - non-determinism means it may not. Sample the same call repeatedly and measure the failure rate; an intermittent 5% failure is a different problem from a deterministic one and needs a different fix.
  • “How do you stop it recurring?” - every incident becomes an eval case. Otherwise a prompt change that fixes today’s problem silently reopens last month’s, and nothing catches it.
  • “How would you set severity?” - not on the uptime scale. In a regulated domain a confidently wrong answer outranks unavailability, because an outage is an inconvenience and a fabricated figure is a liability.
Previous
Cost attribution