AI & ML / Guardrails & safety / 06_model_risk_management.md

Model risk management for LLM systems

Updated 6 interview angles 5 min read source
On this page7
  1. The framework you will be asked about
  2. What the model actually is, for an agent
  3. The three validation questions
  4. What an engineer actually builds
  5. The boundary worth stating
  6. Related
  7. Interview angle

Model risk management for LLM systems

In a bank, a model that makes decisions is a regulated object. Model risk management is the framework governing it, and an engineer working on agents in financial services is expected to know why the controls exist even without being a validation specialist.

The framework you will be asked about

SR 11-7 is the reference - US Federal Reserve and OCC supervisory guidance from 2011 on model risk management. The UK equivalent is the PRA’s SS1/23, published 2023 and effective from 2024. The EU AI Act, in force since 2024 and applying in phases, adds obligations for high-risk AI systems that overlap heavily with these.

Two definitions carry the whole framework:

  • Model - a quantitative method that processes input into an estimate. Deliberately broad, and an LLM in a decision path qualifies.
  • Model risk - the risk of loss from decisions based on incorrect or misused model output. Note misused: a correct model applied outside its intended use is still model risk.

The three lines of defence

Line Who Owns
First Developers and owners Building and documenting
Second Independent validation Challenging it
Third Internal audit Checking the process

The load-bearing word is independent. Validation cannot report to the team that built the model, and “we tested it ourselves” does not satisfy a second line. In practice this means your evaluation artefacts must be legible to someone who did not write the system.

What the model actually is, for an agent

This is the question that separates a considered answer from a recited one.

Classical model risk assumes a stable artefact: weights, an input schema, a documented method. An agent has no such artefact. Its behaviour is produced by a combination:

text 
system prompt   +  model version
tool definitions+  retrieval corpus
memory policy   +  decoding params

Change any one and behaviour changes, often without any code change. A vendor silently updating a model version alters a system that passed validation.

So the engineering requirement is that every component that affects behaviour is versioned and recorded with each run, and that the bundle - not the model alone - is what gets validated.

python 
RUN = {
    "prompt_sha": sha256(system_prompt),
    "model": "vendor-model-2026-05",
    "tools": sorted(t.name for t in tools),
    "corpus_version": index.version,
    "temperature": 0.0,
}

Pin the model version explicitly rather than an alias that floats. An alias means your production system can change without a deploy, which is the thing the framework exists to prevent.

The three validation questions

Validation asks the same three things of any model, and they translate cleanly:

Conceptual soundness. Is this approach right for the task? For an agent: why an LLM rather than rules, what the failure modes are, and where the boundary of intended use sits.

Ongoing monitoring. Is it still working? Drift here is not only data drift - the corpus changes, the vendor changes the model, and user behaviour changes. See Monitoring and drift.

Outcomes analysis. Do the outputs match reality? This is where a golden dataset and regression suite become the evidence, not a nice-to-have - see The eval and observability tooling.

Probabilistic systems and the reproducibility problem

A validator will ask how you test something non-deterministic. The honest answer has three parts: set temperature to zero and pin seeds where the vendor supports it, accept that this reduces rather than removes variance, and therefore evaluate distributions rather than single runs - run the suite N times and assert on pass rates with a threshold, not on exact strings.

Saying “we set temperature to 0 so it is deterministic” is wrong and an interviewer in this space will know it.

What an engineer actually builds

The controls that land in code:

  • A model inventory entry. The system is registered, with an owner, an intended use, and its limitations written down.
  • Traceability per decision. For any output, reconstruct which prompt, model, retrieved documents and tool calls produced it. This is what makes behaviour auditable, and it is the main reason tracing is mandatory rather than useful here.
  • Acceptance thresholds defined before deployment, so a release decision is a comparison rather than a judgement call.
  • A human approval step on consequential actions, with the approval recorded. See Agents and orchestration.
  • A documented fallback for when the model is unavailable or its confidence is low - refusing is a valid, and often required, output.

The boundary worth stating

Model risk management is not the same as guardrails. Guardrails are runtime controls on a single interaction - injection filtering, PII redaction, output validation. Model risk management is the governance lifecycle around the system as a whole: approval, documentation, independent challenge, monitoring, and periodic revalidation.

You need both, and an interviewer asking about a regulated deployment is usually checking whether you conflate them.

Interview angle 6

  • “What is model risk?” - the risk of loss from decisions based on incorrect or misused model output. Misuse matters as much as error: a correct model applied outside its intended use is still model risk, which is why documented intended use is a control.
  • “What is SR 11-7 and who validates a model?” - the 2011 Fed and OCC guidance that most bank frameworks follow, with SS1/23 as the UK equivalent. Validation must be independent of the build team, so your evaluation evidence has to be legible to someone who did not write the system.
  • “What is ‘the model’ when the system is an agent?” - not the weights. Behaviour comes from the prompt, model version, tool definitions, retrieval corpus, memory policy and decoding parameters together, so the bundle is what gets versioned, recorded per run, and validated. Pin the model version rather than a floating alias, or production changes without a deploy.
  • “How do you validate something non-deterministic?” - reduce variance with temperature zero and pinned seeds, then evaluate distributions rather than single runs: execute the suite repeatedly and assert on pass rates against a threshold. Claiming temperature zero makes it deterministic is wrong.
  • “How do you make agent behaviour auditable?” - full-trace capture, so any output can be reconstructed to the prompt, model version, retrieved documents and tool calls that produced it. That is why tracing is a requirement in a regulated deployment rather than a convenience.
  • “How is this different from guardrails?” - guardrails are runtime controls on one interaction; model risk management is the governance lifecycle around the system - approval, documentation, independent challenge, monitoring and revalidation. Conflating them is the common mistake.
Previous
Model governance and responsible AI