Evaluating document extraction

Extraction is the one LLM workload where evaluation is easy in principle — there is a right answer for every field — and it is still where most projects are weakest, because nobody built the labelled set. This note is the metrics, the golden set, and what to watch once the labels stop arriving.

The metrics, by field type

Normalise before comparing. 2024-03-05, 05/03/2024 and 5 March 2024 are the same date; 1.234,56 and 1234.56 the same amount; NFKC Unicode and whitespace differences are not errors. The comparison runs on the typed, normalised value — the same normaliser the pipeline uses (Structured extraction from documents).

def anls(pred: str, gold: str, threshold: float = 0.5) -> float:
    """Average Normalised Levenshtein Similarity, the DocVQA metric."""
    pred, gold = pred.strip().lower(), gold.strip().lower()
    if not pred and not gold:
        return 1.0
    dist = levenshtein(pred, gold)
    sim = 1 - dist / max(len(pred), len(gold))
    return sim if sim >= threshold else 0.0      # below threshold counts as wrong

Straight-through rate — the share of documents where every required field is correct — is the number the business understands, and it is brutal: 98% per-field accuracy over 20 fields is a 67% straight-through rate. Report both; the per-field view says what to fix, the document view says what it costs.

Errors are not all the same

A model that returns null for a smudged total and one that returns the wrong total both score zero on exact match, and they are not the same failure. Classify every miss:

def classify(pred, gold) -> str:
    if pred is None and gold is None: return "correct"
    if pred == gold:                  return "correct"
    if pred is None:                  return "missing"       # safe failure
    if gold is None:                  return "hallucinated"  # invented
    return "wrong"                                           # read badly

Then weight by business cost. A missing value goes to a reviewer; a hallucinated or wrong total that passes validation gets paid. The single most useful chart in a document project is hallucinated + wrong, per field, per vendor — it tells you where the validation ladder needs another rung and where the fine-tune data should come from.

The golden set

• Size. 200–500 documents for a first set; at 200 documents a 90% straight-through rate carries roughly ±4 points of uncertainty, which is enough to rank models and not enough to certify one. Grow it from the review loop.
• Stratify by vendor or template, language, scan quality, handwriting, page count. The tail is where models differ; a random sample under- represents it.
• Double-annotate a subset and adjudicate disagreements. Annotator agreement is the ceiling on measurable accuracy; if two humans agree on a field 92% of the time, a model at 94% is noise.
• Hold it out from everything — training data, few-shot examples in the prompt, and the vendor templates the training set was drawn from. Split by document source, not by document.
• Version it. Gold changes when a labelling error is found; a metric that moved because the labels moved is not a regression.

The pilot builds it (Sizing GPUs against an SLA and a budget), and it is the first artifact that survives a model change.

Comparing two models

Same documents, paired. Report per-field accuracy for both with a bootstrap confidence interval over documents, and count the documents where they disagree — a McNemar test on those is the honest “is B better” answer.

Report tokens per page and latency next to accuracy, because the usual result is “B is 1 point better and costs twice as much”, and that is a decision, not a win. A model change, a prompt change and an engine upgrade all get the same run; the suite lives in CI.

Once the labels stop

Auto-approved documents produce no labels. Three proxies keep the metric alive:

• The review rate and the correction rate, per field and per vendor. Reviewer edits are live labels; their rate is a drift alarm.
• A random audit — a fixed few percent of auto-approved documents sent to review anyway. Without it you cannot measure the silent error rate, which is the one that costs money.
• Distribution monitors — validation-failure rate, mean logprob, image token count per page, share of nulls. A new vendor template or a scanner change shows up here days before anyone complains.

Do not use an LLM as the judge for extraction. There is a right answer; compare to it. A judge is defensible only for the free-text fields where no exact comparison exists, and even there normalised edit distance is usually enough.

Where it breaks in production

• Eval on the vendors you trained on. 98% in the notebook, 90% live. Split by source.
• Per-field accuracy reported alone. The straight-through rate is what the client experiences; leading with 98% sets up the disappointment.
• Labels with one annotator. The model is “wrong” on the fields the labeller got wrong, and nobody finds out.
• Gold as a spreadsheet. Unversioned, edited in place, disconnected from the document hashes. Store it like data.
• No audit of auto-approvals. The pipeline’s reported accuracy drifts from its real accuracy and the gap is invisible until the audit that finance runs.

Related

• Building eval sets - the general practice.
• LLM as judge - why not here.
• Precision, recall and F1 - precision and recall, which the list metrics are.
• Monitoring and drift - the monitors, generally.
• Evaluating vision models and pipelines - when the output is boxes or masks.