AI & ML / Evaluation / 03_llm_as_judge.md

LLM as judge

Updated 8 interview angles 8 min read source
On this page8
  1. The mechanism: why the biases are all about surface
  2. Pairwise beats scoring
  3. Writing a rubric
  4. Validate the judge
  5. Where it breaks in production
  6. Alternatives worth knowing
  7. Related
  8. Interview angle

LLM as judge

Article.

Using a model to score model output. Necessary, because subjective quality has no other scalable measure — and unreliable in specific, well-documented ways you are expected to name.

The single most useful framing, and the one that predicts every bias below: a judge does not measure quality. It predicts what a rater would say.

The mechanism: why the biases are all about surface

A judge is a language model conditioned on a rubric, producing a score token. Whatever correlated with “good” in its training data will shift that token — and in human preference data, the reliable correlates of a highly-rated answer are length, confident phrasing, and clean structure.

So the judge is not failing when it prefers a longer answer. It is doing exactly what it was trained to do, which is reproduce a rater’s response, including the parts of that response driven by presentation rather than substance.

That reframes the mitigation list from “tricks” into “controlling for the features it is known to key on”:

Bias Effect Mitigation
Position prefers whichever came first swap order, require agreement
Verbosity prefers longer answers control length; score conciseness
Self-preference prefers its own family’s text judge with a different model
Sycophancy agrees with the prompt’s framing never reveal which answer is yours
Formatting prefers confident, tidy prose rubric must name substance
Scale compression clusters on 7-8 of 10 small discrete scale, or pairwise

Position and verbosity are the two to name unprompted. They are large, cheaply mitigated, and the fastest signal that someone has actually run one.

Pairwise beats scoring

Absolute scores drift between runs and cluster in a narrow band. “Which of these two is better” is a question models answer far more consistently, because it removes the need to calibrate against an imagined scale.

python 
def compare(query: str, a: str, b: str) -> str:
    """Run both orderings; disagreement is an honest tie."""
    first  = judge(query, a, b)     # -> "A" | "B" | "tie"
    second = judge(query, b, a)     # order swapped

    if first == "A" and second == "B":
        return "a"                  # both orderings prefer a
    if first == "B" and second == "A":
        return "b"
    return "tie"

Treating cross-ordering disagreement as a tie is the move that matters: it converts position bias into “too close to call” instead of a coin flip that looks like a result.

Pairwise also composes with the paired statistics in Building eval sets — wins, losses and ties over the same cases is exactly the shape McNemar wants.

Writing a rubric

python 
RUBRIC = """Score the ANSWER against the QUESTION and CONTEXT.

Criteria, in priority order:
1. Grounded: every factual claim appears in CONTEXT. An unsupported claim is
   disqualifying regardless of how plausible it sounds.
2. Complete: addresses every part of the question.
3. Concise: no padding, no restating the question.

Score:
  2 = grounded and complete
  1 = grounded but incomplete, or minor unsupported detail
  0 = an unsupported claim, or fails to address the question

Return JSON: {"reason": "<one sentence>", "unsupported": [...], "score": 0|1|2}
"""

Four things make that work:

  • A 3-point scale. Models do not reliably separate 7 from 8, and you do not need them to.
  • Priority order, so the judge knows what dominates when criteria conflict.
  • One disqualifying criterion, stated plainly.
  • reason before score in the output. Generation is left to right, so the reasoning conditions the score rather than rationalising one already emitted. Swapping those two fields measurably changes results.

The reason field is also how you audit the judge, and how you find out it has been scoring something other than what you meant.

Validate the judge

The step that gets skipped. A judge is a model with its own error rate.

python 
from sklearn.metrics import cohen_kappa_score

kappa = cohen_kappa_score(human_scores, judge_scores)   # 50-100 labelled cases

Kappa corrects for chance agreement, which raw accuracy does not — and that matters here, because a judge that scores almost everything 2 will look accurate against a set that is mostly good.

Above about 0.6 is reasonable agreement; below 0.4 the judge is measuring something other than what you intended and its numbers are decoration.

If agreement is poor: sharpen the rubric, move to pairwise, use a stronger judge model, or accept that the dimension needs human review. Re-validate whenever the rubric or the judge model changes — a judge is a versioned component, not a constant.

Where it breaks in production

Optimising against the judge

The most important failure, and the least discussed.

If you use the judge to select prompts — try twenty variants, keep the highest-scoring — you are no longer measuring quality. You are running a search for text that the judge scores highly, and the judge’s known biases are the cheapest way to score highly.

text 
generate variants -> score -> keep the best -> repeat
                         |
            optimises for verbosity and confidence,
            which is what the judge rewards

The result is a prompt that reliably produces longer, more assertive answers and performs no better with users. Goodhart’s law, arriving on schedule: a measure used as a target stops being a good measure.

The defence is to hold out a validation set the optimisation never sees, and to confirm any judge-driven win against human labels or a production metric before shipping it.

Judge drift

The judge model updates, every score shifts two points, and nothing in your system changed. Pin the judge version, record it alongside every result, and treat a judge upgrade as an event that requires re-baselining — not as a silent dependency bump.

Rubric changes are the same problem: a score is only comparable to another score produced under the same rubric version. Version the rubric with the results.

Cost arithmetic

A judge call per case sounds cheap until you multiply it out:

text 
200 cases x 2 orderings x 2 variants
    = 800 judge calls per experiment

Practical controls: short rubrics and small scales keep output tokens down, eval is not latency-sensitive so use a batch API, run a cheap judge on the smoke set and a strong one on release gates, and cache on (rubric_version, judge_version, output_hash) so re-running a suite does not re-pay for unchanged outputs.

A judge that never disagrees

High agreement with humans on a set where everything is good tells you nothing. Check the distribution of judge scores, not just agreement: if 95% of cases score 2, the judge is not discriminating and your suite has no headroom to detect a regression.

Alternatives worth knowing

  • NLI / entailment models for groundedness. Decompose the answer into claims and check each against the retrieved context. Smaller, cheaper, and often more reliable than a frontier model eyeballing it — the most underused option here.
  • Reference-based metrics (BLEU, ROUGE, BERTScore). Surface overlap, poorly correlated with quality for open-ended generation. Occasionally right for translation or tightly-constrained summarisation.
  • Human review on a sample. The ground truth everything else is validated against. Expensive, so sample rather than exhaust.

The trade not taken

The strongest move is often to avoid needing a judge at all by changing what you ask the system to produce.

An answer required to cite the chunk supporting each claim can be checked with code: does the citation exist, and does the cited chunk contain the claim? That converts “is this good” into a deterministic check plus a cheap entailment call.

Every dimension you can move out of the judge’s hands is one that stops drifting, stops costing a model call, and stops being something you have to validate. Reach for the judge for what genuinely remains subjective — tone, helpfulness, coherence — and not for anything a rule could decide.

Interview angle 8

  • “How do you evaluate subjective output quality?” — deterministic checks for everything code can verify, then a judge for what remains, with a specific rubric and a small discrete scale. Validate the judge against human labels before trusting a single number from it.
  • “What biases does an LLM judge have?” — position, verbosity, self-preference, sycophancy, formatting, and scale compression.
    • Follow-up: “Why does it have those in particular?” — because it predicts what a rater would say rather than measuring quality, and in preference data the reliable correlates of a good answer are length, confidence and clean structure. The biases are the model doing its job.
  • “Scoring or pairwise?” — pairwise. Absolute scores drift and cluster; relative comparison is far more consistent. Run both orderings and treat disagreement as a tie, which turns position bias into an honest tie instead of a coin flip.
    • Follow-up: “What do you do with the wins and losses?” — feed them into a paired test. Wins, losses and ties over the same cases is exactly the shape McNemar wants, and it detects far smaller differences than comparing two independent pass rates.
  • “How do you know the judge is any good?” — human-label 50-100 outputs and measure Cohen’s kappa, which corrects for chance. Below about 0.4 it is measuring something else. Also check the score distribution: a judge that gives almost everything full marks agrees with humans and detects nothing.
  • “You used the judge to pick the best of twenty prompts. Any concern?” — yes, that is optimising against the measure. The cheapest way to score well is to exploit the judge’s verbosity and confidence bias, so you get longer, more assertive answers that are no better. Hold out a validation set and confirm against human labels or a production metric.
  • “Your scores moved and the code did not.” — the judge model or the rubric changed. Both are versioned components; pin them, record them with every result, and re-baseline deliberately after an upgrade.
  • “Cheaper way to check groundedness?” — decompose into claims and run a dedicated entailment model against the retrieved context. Better and cheaper than a general judge for that specific question.
  • “When would you not need a judge?” — when you can change the output contract. Require a citation per claim and the check becomes code: does the citation exist, and does that chunk support the claim.
Previous
Building eval sets