Structured extraction from documents

The production shape of “use a VLM to read invoices, forms and scans into a database” is a pipeline in which the model is one stage, and most of the engineering is around it: preprocessing that decides what the model sees, validation that decides what to believe, and a review loop that turns mistakes into labels. Interviewers for this role probe the stages, not the prompt.

The pipeline

upload ─▶ queue ─▶ preprocess ─▶ classify ─▶ extract
 (S3)   (SQS /    (rotate, dpi,  (doc type)  (VLM +
        Celery)    split pages)               schema)
                                                │
   store ◀─── route ◀─── validate ◀─────────────┘
 (Postgres) (auto / review) (pydantic + rules)

Every stage writes its output somewhere durable and is idempotent on the document’s content hash, so a crash mid-batch re-runs the stage, not the customer’s upload. The queue and worker mechanics are the ordinary ones — Consumer idempotency — with one difference: the expensive stage runs on GPU workers that want batches of pages, not one document at a time.

The schema is the product

from enum import StrEnum
from pydantic import BaseModel, Field

class Currency(StrEnum):
    EUR = "EUR"; USD = "USD"; GBP = "GBP"

class Extracted[T](BaseModel):
    value: T | None                       # null is legal, and preferred
    quote: str | None = None              # verbatim source text
    page: int | None = None

class Invoice(BaseModel):
    invoice_number: Extracted[str]
    issue_date: Extracted[str] = Field(description="ISO 8601 as printed")
    currency: Extracted[Currency]
    total: Extracted[float]
    line_items: list[LineItem] = Field(max_length=200)

SCHEMA = Invoice.model_json_schema()      # -> constrained decoding

Three decisions are baked in. Every value is nullable, because a schema that demands a number gets an invented one. Every value carries its quote and page, so a hallucination can be caught by checking the quote against the page text, and a reviewer can see where the number came from. And the schema is versioned in the repository and stored with each row, because the day you add a field, every old row must still parse.

Do not ask the model to rate its own confidence; self-reported confidence is poorly calibrated. Token logprobs from the engine, calibrated against your labels, are the confidence signal (VLM failure modes in production).

Preprocessing decides what the model sees

• Text layer first. A PDF with real text goes to a text extractor and a text model; only pages with no or broken text are rendered for the VLM. It is cheaper by an order of magnitude and more faithful on characters.
• Render at 200–300 dpi, detect orientation and rotate, split into pages, cap the pixel budget per page deliberately rather than by the processor default (How a VLM sees an image).
• Classify before extracting. A cheap pass — text heuristics, or the VLM at a low pixel budget — labels each page (invoice, terms, blank, attachment) so the expensive pass only runs on pages that carry fields.
• Hash and dedupe. The content hash is the idempotency key and catches the same invoice uploaded three times.

The call

import base64

def extract(client, page_png: bytes, doc_type: str) -> dict:
    data_url = "data:image/png;base64," + base64.b64encode(page_png).decode()
    resp = client.chat.completions.create(
        model="qwen3-vl-8b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPTS[doc_type]},
            {"role": "user", "content": [
                {"type": "image_url", "image_url": {"url": data_url}},
                {"type": "text", "text": "Extract the fields. Use null for "
                                         "anything not clearly readable."},
            ]},
        ],
        response_format={"type": "json_schema",
                         "json_schema": {"name": "invoice", "schema": SCHEMA}},
        temperature=0, max_tokens=2000, logprobs=True,
    )
    return resp.choices[0].message.content

The system prompt is stable per document type so the prefix caches; the schema is enforced by the engine, not requested; max_tokens is derived from the largest legitimate output plus margin, so a repetition loop is cut off rather than billed. Serving-side detail in Serving VLMs.

Post-processing and validation

The model returns strings shaped like the page. Post-processing turns them into typed values, and validation decides whether to believe them.

import re
from decimal import Decimal

def parse_amount(s: str) -> Decimal:
    """'1.234,56' and '1,234.56' both -> 1234.56; last separator is decimal."""
    s = re.sub(r"[^\d,.\-]", "", s)
    if s.count(",") and s.count("."):
        dec = s[max(s.rfind(","), s.rfind("."))]
    else:
        dec = "," if re.search(r",\d{2}$", s) else "."
    return Decimal(s.replace("." if dec == "," else ",", "").replace(dec, "."))

def iban_ok(iban: str) -> bool:
    s = re.sub(r"\s", "", iban).upper()
    s = s[4:] + s[:4]
    return int("".join(str(int(c, 36)) for c in s)) % 97 == 1

The validation ladder, cheapest first:

1. Shape — the schema, already guaranteed by constrained decoding.
2. Format — dates parse, IBAN mod-97, VAT number pattern, currency in the enum. Pure functions; a failure is a definite error.
3. Evidence — each quote is found (fuzzily) in the OCR text of its page. A missing quote is the strongest single hallucination signal.
4. Business rules — arithmetic, date ordering, vendor exists in master data, total within the PO’s tolerance.
5. Confidence — mean logprob of the value tokens against a per-field threshold learned from the golden set.

A document that passes all five is auto-approved; one that fails any goes to review with the failing rule named. The review rate is the product metric — the number the client sees on the dashboard.

The review loop is the data engine

Every correction a reviewer makes is a labelled example. Store the corrected document, the original prediction and the reviewer’s identity; that table is your regression suite, your calibration set for the confidence thresholds, and — once it has a few thousand rows — the training set for a fine-tune that moves the review rate (Fine-tuning VLMs).

Auto-approval thresholds are set per field and per document type from this data, never by feel.

Where it breaks in production

• Partial success reported as success. Page 17 of 40 timed out; the merge produced a document with a missing table; the row looks complete. Fan-out with a fan-in that counts, and a status that distinguishes complete from partial.
• Retries that re-extract. A non-idempotent worker retried after a broker hiccup extracts twice and, on an invoice, pays twice. Content-hash keys, and the write is an upsert.
• Schema change with no version. Field renamed; every stored row now fails to parse in the reader. Version in the row, migrate forward.
• The paraphrased quote. The model “quotes” a cleaned-up version and the evidence check passes on fuzziness alone. Tighten the match for numeric fields to exact.
• Image bytes in logs. A debug line dumping the request leaks the customer’s document into the logging platform. Log hashes and sizes.
• Review queue without a cap. Thresholds too tight, reviewers drown, and the “human in the loop” is a backlog. Watch the review rate and the queue age together.

Related

• Constrained decoding on self-hosted models - how the schema is enforced.
• Evaluating document extraction - measuring the pipeline, field by field.
• Structured Document Pipelines — Parsing, Tables, OCR, Extraction - the text-side parsing stack this shares stages with.
• Kafka vs RabbitMQ — choosing a message broker - choosing the queue.
• PII, privacy and the EU AI Act - what leaves your boundary with the page.