AI & ML / Guardrails & safety / 05_governance_and_responsible_ai.md

Model governance and responsible AI

Updated 6 interview angles 6 min read source
On this page8
  1. The artefacts that constitute it
  2. Risk tiering decides how much process applies
  3. Human oversight has to be meaningful
  4. Auditability is a retention and correlation problem
  5. Change control for things that are not code
  6. Saying it well
  7. Related
  8. Interview angle

Model governance and responsible AI

The vocabulary regulated employers use. The engineering underneath is in the rest of this folder — injection defence, PII handling, red teaming, fairness — but a bank’s job ad asks for “model governance” and “responsible AI”, and answering with implementation detail alone reads as junior.

Governance is the answer to one question: who decided this model could be used for this, on what evidence, and how would you prove it later?

The artefacts that constitute it

Artefact Records
Model card what it is, limits, evaluation
Risk register what could go wrong, and controls
Data lineage where training and RAG data came from
Decision log why this model, who approved
Eval history scores over time, per version

None of these are exotic. What makes governance real is that they are maintained artefacts rather than a launch document, because the audit question is not “did you think about this” but “show me the version from March”.

The model card

The one to be able to describe. It is a short document covering:

  • Intended use, and explicitly out-of-scope use. The second half is what protects you when someone repurposes the system.
  • Training or grounding data, and its known gaps.
  • Evaluation results, including sliced by group — see Fairness and bias.
  • Limitations and failure modes in plain language.
  • Owner and review date.

It is a file in the repository, reviewed like code:

yaml 
system: loan-pre-screen
owner: risk-platform
review_by: 2027-02-01
intended_use: Rank applications for human review.
out_of_scope:
  - Automated decline without review
  - Any decision on an existing customer
evaluation:
  suite: evals/loan_v4
  sliced_by: [age_band, region]
limitations:
  - Trained on UK applications only.

For an LLM application you are usually not writing a card for the model — the provider did — but for the system: the prompt, the retrieval corpus, the guardrails and the evaluation are yours, and they are what changes behaviour.

Risk tiering decides how much process applies

The EU AI Act’s structure is worth knowing because everyone borrows it:

Tier Examples Obligation
Unacceptable social scoring banned
High risk credit, hiring, medical heavy
Limited chatbots disclosure
Minimal spam filter none

High-risk brings data governance, technical documentation, logging, human oversight and post-market monitoring. Those obligations phase in through 2027-2028; Article 50 transparency and GPAI duties are already in force as of 2026-08.

The practical consequence for an engineer: credit scoring and CV screening are high risk, so an AI feature touching either changes what the team must produce, not just what it must be careful about. Saying that unprompted in a FinTech interview is a strong signal.

Human oversight has to be meaningful

“A human reviews it” is the control everyone claims and few implement well. The failure is rubber-stamping: a reviewer approving 200 items an hour is not oversight, and an auditor will notice the throughput.

What makes it real:

  1. The reviewer can see why the model decided — see Interpretability.
  2. They can override, and overrides are recorded as training signal.
  3. The volume is compatible with actually reading.
  4. Disagreement rate is monitored — if it is near zero, the human is not adding anything.

That last metric is the one that separates a considered answer.

Auditability is a retention and correlation problem

To reconstruct a decision months later you need, joined by one id: the input, the retrieved context, the prompt version, the model version, the parameters, the output, and who reviewed it.

Two tensions worth naming rather than resolving:

  • Retention versus data minimisation. Auditability wants long history; GDPR wants the minimum necessary. The usual settlement is a longer-lived structured audit record and shorter-lived full payloads.
  • Right to erasure versus immutable logs. Pseudonymise in the audit trail so a deletion request can be satisfied without destroying the record.

See PII, privacy and the EU AI Act.

Change control for things that are not code

The governance gap specific to LLM systems: the behaviour-defining artefacts are not all in your repository.

Changes behaviour Usually versioned?
Application code yes
Prompt often not
Retrieval corpus rarely
Model version rarely pinned
Guardrail config rarely

Every row that is not versioned is a way for behaviour to change with no review and no rollback. The fix is to make all five one reviewable object:

python 
@dataclass(frozen=True)
class Release:
    code_sha: str
    prompt_version: str      # not a file on disk
    corpus_snapshot: str     # an index id, dated
    model: str               # pinned, never an alias
    guardrail_config: str

If a rollback cannot restore every field, the row it cannot restore is the one that will cause the incident. Pinning the model, versioning prompts as artefacts and versioning the corpus is what makes “what changed?” answerable — the same requirement that Incident response for AI systems depends on.

Saying it well

“We treat the prompt, the corpus and the model version as versioned artefacts with a review step, we keep sliced evaluation results per version, and every decision is reconstructable from the trace for the retention period. High-risk use cases get documented human oversight with a monitored override rate.”

That paragraph is what “model governance” means operationally, and it is mostly things you would want anyway.

Interview angle 6

  • “What does model governance mean to you?” - being able to answer who decided this model could be used for this, on what evidence, and prove it later. Concretely: model cards, a risk register, data lineage, a decision log and evaluation history, maintained as living artefacts rather than a launch document.
  • “What’s in a model card?” - intended use and explicitly out-of-scope use, the data and its gaps, evaluation results sliced by group, limitations in plain language, and an owner with a review date. For an LLM application you write it for the system — prompt, corpus, guardrails — not the model, since the provider wrote that one.
  • “Which use cases are high risk?” - credit scoring, hiring and medical, under the EU AI Act’s tiering. That changes what the team must produce — data governance, documentation, logging, human oversight, post-market monitoring — not merely how careful it is.
  • “What makes human oversight real rather than nominal?” - the reviewer can see why the model decided, can override, has a volume compatible with actually reading, and the disagreement rate is monitored. A near-zero disagreement rate means the human is rubber-stamping.
  • “What’s the governance gap specific to LLM systems?” - the behaviour-defining artefacts are not all in the repository. Prompts, the retrieval corpus, the model version and guardrail config change behaviour and are usually unversioned, so behaviour changes with no review and no rollback.
  • “How do you reconcile audit retention with GDPR?” - a longer-lived structured audit record with pseudonymised identifiers, and shorter-lived full payloads. That satisfies erasure requests without destroying the ability to reconstruct a decision.
Previous
Red teaming