AI & ML / Agents & orchestration / 07_agent_failure_modes.md

Agent failure modes

Updated 8 interview angles 9 min read source
On this page9
  1. The mechanism: the transcript is the whole world
  2. Loud failures
  3. Silent failures
  4. Detection, not prevention
  5. Cost, latency and effects
  6. Security
  7. Testing something non-deterministic
  8. Related
  9. Interview angle

Agent failure modes

Article.

Every agent failure list is the same list. What separates someone who has run an agent from someone who has read about them is not the taxonomy — it is knowing which failures announce themselves and which ones do not, because they need completely different treatment.

The loud ones you guard against with limits. The silent ones you cannot guard against at all; you can only detect them after the fact, and most teams never build the detection.

The mechanism: the transcript is the whole world

Almost every failure below follows from one property.

An agent knows nothing except what is in its message list. Not what your tool meant, not whether a call actually happened, not whether a conclusion three steps back was justified. Everything is text in a sequence, distinguished only by a role field.

text 
[user]      find the latest order and refund it
[assistant] tool_call: search_orders(email="a@b.com")
[tool]      ""        <- the tool raised, returned ""
[assistant] "I could not find any orders."

The tool crashed. What arrived in the transcript was an empty string, which is exactly what “no results” looks like. The agent read it correctly and reported it honestly, and the answer is wrong.

Once you hold that model, the failure list stops being a list and becomes consequences: anything you fail to put in the transcript, the agent cannot know; anything you put in it, the agent will treat as true.

Loud failures

These trip a limit or raise an exception. They are the easy half — you need the guard, but nothing subtle is going on.

Failure Guard
Infinite loop step limit, plus loop detection
Cost runaway token and cost budget per run
Stuck run wall-clock timeout
Malformed arguments validate before executing
Context overflow bound tool output at the tool

Loop detection has one detail worth keeping:

python 
key = (call.tool, json.dumps(call.args, sort_keys=True))
seen[key] += 1

if seen[key] > 2:
    result = (
        f"You have called {call.tool} with these arguments {seen[key]} times "
        "and received the same result. Try a different approach, or explain "
        "what is blocking you."
    )

sort_keys=True is load-bearing. Argument dicts serialise in whatever order the model emitted them, so without sorting, three identical calls hash to three different keys and the detector never fires.

And feed the detection back rather than aborting — the model usually recovers when told it is stuck, and aborting converts a recoverable state into a failure.

Silent failures

These produce a confident, plausible, wrong answer, and nothing in your metrics moves. This is the half that matters.

The tool that fails quietly

The most common one in production, and the one an experienced interviewer will recognise instantly.

python 
# The bug. Looks defensive; is not.
async def search_orders(email: str) -> list[dict]:
    try:
        return await api.search(email=email)
    except Exception:
        return []          # indistinguishable from "no matches"

An empty list and a failed call are the same observation. The agent cannot tell them apart because you did not put the difference in the transcript.

python 
# The fix. The error is information the model can act on.
async def search_orders(email: str) -> dict:
    try:
        return {"results": await api.search(email=email)}
    except TimeoutError:
        return {"error": "search timed out", "retryable": True}
    except NotFoundError:
        return {"results": [], "note": "no orders for this address"}

Now the agent retries the timeout and reports the genuine empty result. The distinction costs four lines and is the difference between an agent that says “nothing found” and one that says “the search is down”.

Confabulated tool results

The model sometimes writes what a tool would have returned instead of calling it — more likely with weaker models, long transcripts, and tools it has called successfully several times already.

It follows from the mechanism: in the transcript, a previous tool result and the model’s own text differ only by a role tag, so the pattern “call, then result” is one the model can complete without your dispatcher ever running.

You detect it by reconciling claims against your own logs:

python 
called = {(s.tool, s.args_hash) for s in trace.tool_invocations}
claimed = extract_tool_references(final_answer)

if unbacked := claimed - called:
    log.error("answer cites uninvoked tools",
              extra={"unbacked": unbacked})

The general principle: your execution log is ground truth, the transcript is testimony. Anything the answer asserts should be traceable to something you actually ran.

Compounding errors

A wrong intermediate conclusion at step 3 becomes input at step 4 with exactly the same standing as a verified tool result. By step 8 the agent is reasoning confidently from a false premise, and nothing in the transcript marks it as less trustworthy than anything else.

Mitigations, in order of effectiveness:

  • Ground each step in tool output, not in prior reasoning. Prefer re-reading a value to remembering it.
  • Keep chains short. The failure probability compounds per step, so a 15-step task is not 50% more risky than a 10-step one.
  • Verify at the boundary. Before an action with effects, re-check its preconditions with a tool call rather than trusting the transcript.

Overconfidence when it failed

The agent could not complete the task and produces a confident answer anyway, because a fluent answer is what the training distribution rewards and “I don’t know” is rarely what it saw.

The fix is structural rather than a prompt instruction: make the gap a required field, so not-knowing is a value the schema can hold.

python 
class Answer(BaseModel):
    result: str | None
    confidence: Literal["high", "medium", "low"]
    unresolved: list[str] = Field(
        description="Anything you could not verify. Empty if all verified."
    )

An empty unresolved on a run whose trace shows three failed tool calls is itself a signal worth alerting on.

Detection, not prevention

The practical consequence of the split above: your guards handle the loud half, and you need a separate mechanism entirely for the silent half.

Signal Catches
Uninvoked tools cited confabulation
Empty unresolved overconfidence
Tool error rate a quietly-failing dependency
Step-limit exhaustion tasks drifting out of reach
Answer length collapse degradation before users notice

Step-limit exhaustion is the best single leading indicator. It moves before task success falls and well before anyone complains, because the agent tries harder for longer before giving up.

Cost, latency and effects

Cost is superlinear in steps — each step re-sends the whole transcript, so total input tokens grow with the square of the step count. Cap tokens and money per run, not just steps; see The agent loop.

Latency is additive and visible. Ten sequential steps at two seconds each is twenty seconds, and users leave. Stream progress, parallelise independent calls, and ask whether a fixed workflow would have done.

Partial completion is the hard one. Failure at step 7 with steps 1-6 already applied to the world is not undone by aborting. Either make effects idempotent and retryable, or design so effects commit only at the end — see Idempotency keys.

Security

Three that are architectural, not promptable:

  • Prompt injection reaching tools. Untrusted content instructs the agent to misuse its own permissions. Mitigate with least privilege per agent, read-only separated from write-capable, approval gates on consequential actions, and authorization enforced server-side rather than by what the model asked for.
  • Exfiltration through a tool that takes a URL. Any tool with an arbitrary destination is a channel out. Restrict egress and audit it.
  • Excessive agency. Tools the task did not need. Scope the tool set per task, not per application.

See Building and securing MCP servers.

Testing something non-deterministic

  • Regression suites over curated tasks, scored on the final outcome rather than the exact trajectory — trajectories vary legitimately.
  • Trajectory assertions for what must hold: “must call verify_identity before refund_order” is both testable and the thing you actually care about.
  • Mocked tools for determinism and speed, with a smaller live suite run separately.
  • Adversarial cases: injection attempts, failing tools, missing data, ambiguous requests.
  • Cost and step budgets as assertions, so a change that doubles spend fails CI rather than arriving on the invoice.

That last one is unusual and worth having — treating cost as a tested property catches regressions no accuracy metric will.

The trade not taken

Every guard above is complexity, and there is a design that needs almost none of it: do not use an agent. A fixed workflow with one model call per stage has no loops to detect, no compounding chain, linear cost, and failures that land in one identifiable stage.

The agent earns its failure surface only when the number of steps genuinely cannot be known in advance. Most systems described as agents are workflows, and saying so is a stronger interview answer than reciting the mitigations.

Interview angle 8

  • “What goes wrong with agents in production?” — split them by whether they announce themselves. Loops, cost runaway and malformed arguments trip a limit and are handled with guards. Silent tool failure, confabulation, compounding errors and overconfidence produce a confident wrong answer with nothing in your metrics moving, and need detection instead.
    • Follow-up: “Why does that distinction matter?” — because guards are useless against the silent half. No step limit catches an agent that confidently reports “no orders found” when the search API was down.
  • “A tool fails and the agent says nothing was found. What happened?” — the tool returned an empty list on exception, so a failure and a genuine empty result are the same observation. An agent knows only what is in its transcript, so the difference has to be put there: return an error object with a retryable flag.
  • “How do you know if a model fabricated a tool result?” — reconcile the final answer against your invocation log. The execution log is ground truth and the transcript is testimony; anything asserted should trace to something you ran.
    • Follow-up: “Why would it do that at all?” — because in the transcript a prior tool result and the model’s own text differ only by a role tag, so “call, then result” is a pattern it can complete without your dispatcher ever running.
  • “Why do agents get worse over a long run?” — a wrong intermediate conclusion enters the transcript with the same standing as a verified tool result, so later steps reason from it confidently. Ground each step in tool output rather than prior reasoning, and re-check preconditions before acting.
  • “How do you stop it inventing an answer when it failed?” — make the gap a required schema field, so “unresolved” is a value it can return rather than an absence. Then alert when that field is empty on a run whose trace shows failed tool calls.
  • “Which metric warns you first?” — step-limit exhaustion rate. It rises before task success falls, because the agent tries longer before giving up.
  • “How do you test something non-deterministic?” — outcome-scored regression suites, trajectory assertions for orderings that must hold, mocked tools for determinism, adversarial cases, and cost and step budgets asserted in CI.
  • “How would you avoid most of this?” — do not use an agent. A fixed workflow has no loop to detect, linear cost, and failures isolated to a stage. The loop is only justified when the step count cannot be known in advance.
Previous
Agent protocols: MCP, A2A and the interop layer