Building eval sets

Article.

The unglamorous work that determines whether everything else means anything. A hundred good cases beat any framework, and the hard part is not collecting them — it is knowing when a number has moved and when it has not.

Most teams get the collecting roughly right and the statistics badly wrong, so this note spends its depth on the second half.

Where cases come from

Start with production logs, sampled stratified across intents and difficulty rather than just the head.

A synthetic-only set tests the system against questions you imagined people would ask. Real queries are shorter, more ambiguous, full of typos and product nicknames, and frequently about something the corpus does not cover — which is exactly the category worth measuring.

What a case looks like

class EvalCase(BaseModel):
    id: str
    query: str
    context: dict = {}                  # user state, conversation history
    expected_chunks: list[str] = []     # for retrieval recall
    must_contain: list[str] = []        # required facts
    must_not_contain: list[str] = []    # forbidden content
    should_refuse: bool = False
    rubric: str | None = None           # only if a judge is needed
    tags: list[str] = []                # intent, difficulty, source

tags matter more than they look. They let you report per segment rather than one aggregate, which is how you notice the system got worse for one intent while the average improved — the single most common way a regression ships.

Coverage that catches real failures

• The head, weighted as it actually occurs.
• The tail — rare but important intents.
• Ambiguous queries where the correct behaviour is to ask a question back.
• Unanswerable queries, where the corpus genuinely does not contain the answer.
• Adversarial — injection, jailbreaks, off-topic.
• Multi-turn follow-ups that are meaningless standalone.

Unanswerable is the category most often missing, and it is the one that matters most for trust. Without it you never measure whether the system admits ignorance or invents an answer, and inventing is what loses users permanently.

Deterministic checks first

Before reaching for a judge, extract everything mechanically checkable:

def deterministic_checks(case: EvalCase, output: str) -> dict[str, bool]:
    return {
        "parses": try_parse_json(output) is not None,
        "schema_valid": validate_schema(output),
        "contains_required": all(s in output for s in case.must_contain),
        "avoids_forbidden": not any(s in output for s in case.must_not_contain),
        "has_citation": bool(CITATION_RE.search(output)),
        "refused": is_refusal(output) == case.should_refuse,
    }

Free, fast, deterministic, and they catch a large share of real regressions. A judge is for what is left over, not for what a regex could have told you — see LLM as judge.

The mechanism: why your score moved

This is the part that separates an eval suite from a number generator.

A bare percentage is not a result

from scipy.stats import beta

def score_ci(passes: int, n: int, conf: float = 0.95) -> tuple[float, float]:
    """Report the interval, never the bare rate."""
    lo = beta.ppf((1 - conf) / 2, passes, n - passes + 1) if passes else 0.0
    hi = beta.ppf(1 - (1 - conf) / 2, passes + 1, n - passes) if passes < n else 1.0
    return lo, hi

On 100 cases an 85% pass rate carries a 95% interval of roughly ±7 points. So an 85% to 88% “improvement” is indistinguishable from having run the same system twice. Teams ship changes that did nothing, constantly, on exactly this.

Paired comparison, and why it is so much stronger

The fix is not always a bigger set. Run both variants on the same cases.

Comparing two independent rates means the noise of both adds up. Comparing paired outcomes cancels the thing that dominates that noise: case difficulty. A hard case is hard for both variants, so it tells you nothing about which is better — and when you compare rates, it is still contributing variance.

Only the disagreements carry signal:

             B pass   B fail
A pass         82       6      <- b
A fail          3       9      <- c
               ^
               only b and c say anything

Eighty-two cases both got right and nine both got wrong are noise-free and uninformative. The question is whether 6-versus-3 is a real asymmetry, which is McNemar’s test:

from scipy.stats import chi2

def mcnemar(b: int, c: int) -> float:
    """b = A passed, B failed.  c = A failed, B passed."""
    if b + c == 0:
        return 1.0
    stat = (abs(b - c) - 1) ** 2 / (b + c)      # continuity-corrected
    return 1 - chi2.cdf(stat, df=1)

With 6 and 3 that is not significant — but the same experiment as two independent rates would need several times the cases to say anything at all. Pair whenever you can, which is whenever both variants can run on the same inputs.

Sizing

100-200 is the sweet spot: large enough that a real change clears the interval, small enough to run in minutes for a few dollars. Below that you are measuring noise; above it you stop running it, which is worse.

Where it breaks in production

The eval set that leaks into the prompt. Few-shot examples chosen from the eval set make the score go up and the system no better. Keep the sets physically separate and assert it in CI:

assert not (set(EVAL_IDS) & set(FEWSHOT_IDS)), "eval cases leaked into prompt"

Cases generated from the documents the system retrieves. An LLM given a chunk and asked to write a question produces one whose answer is that chunk, phrased in its vocabulary. The retriever finds it trivially. You measure 95% and users still complain, because real queries do not use the corpus’s wording — that mismatch is the whole problem retrieval exists to solve.

Deleting failing cases. The suite goes green, the regression ships, and the case that would have caught it next time is gone. Cases are append-mostly; a case that starts failing is the suite doing its job.

Judge drift. The judge model updates, scores move two points, and nothing in your system changed. Pin the judge version and record it with every result, or your longitudinal numbers are meaningless.

Scores without outputs. Storing only pass rates means that when something regresses you cannot read what it actually produced, and you cannot re-score old runs under a corrected rubric. Store the outputs.

The feedback loop

production failure -> triage -> add case -> fix -> passes
                                      |
                                stays forever

This is the mechanism that makes a system improve rather than churn. A complaint that does not become a case will recur.

Weekly triage of production failures into the suite costs about an hour and compounds. It is a process answer rather than a technical one, which is exactly why it lands well in interviews — most candidates describe tooling and few describe the loop.

The trade not taken

Every case is maintenance. A 500-case suite that nobody runs because it takes forty minutes and costs real money is worth less than 60 cases that run on every commit.

And for a genuinely deterministic feature — a classifier with five outcomes, an extractor with a fixed schema — you may need no LLM evaluation at all. Ordinary unit tests with fixed inputs and expected outputs are cheaper, faster and exactly as informative. Reach for an eval suite when the output space is open enough that assertions cannot describe correctness, and not before.

Related

• LLM as judge - grading what deterministic checks cannot
• Why evaluating LLM systems is hard - why this is different from testing
• Online evaluation and experiments - measuring in production