Vision-language models
Open-weight VLMs in production, aimed at document workloads — scans, PDFs, forms — rather than photographs: how the models see, which ones exist and why, how they fail, and the extraction, serving, fine-tuning and evaluation work around them.
| # | File | Covers |
|---|---|---|
| 01 | How a VLM sees an image | encoder–projector vs cross-attention, resolution strategies, the token arithmetic, why counting is weak |
| 02 | Open-weight VLMs: the landscape | Qwen2.5-VL / Qwen3-VL, InternVL, Llama Vision, Pixtral, Gemma 3, the document-OCR VLMs, licences, benchmarks |
| 03 | VLM failure modes in production | hallucinated fields, small print, counting, repetition loops, rotation, multi-page, injection |
| 04 | Structured extraction from documents | the pipeline: schema with evidence, preprocessing, the validation ladder, the review loop |
| 05 | Constrained decoding on self-hosted models | XGrammar / Outlines, vLLM and SGLang structured outputs, the schema subset, how it lowers accuracy |
| 06 | Serving VLMs | vLLM flags for images, KV budget and concurrency, prefill-heavy tuning, the CPU bottleneck |
| 07 | Fine-tuning VLMs | LoRA / QLoRA on the language side, when to unfreeze vision, resolution skew, merge then quantise |
| 08 | Sizing GPUs against an SLA and a budget | three measured numbers, a worked GPU count and cost, latency vs throughput budgets, self-host vs API |
| 09 | Evaluating document extraction | ANLS, TEDS, straight-through rate, the error taxonomy, the golden set, monitoring after labels stop |
| 10 | Layout detection and OCR engines | DocLayout-YOLO, Surya, PaddleOCR, Tesseract, table recognition, the hybrid that ships |
The three things that carry most interviews
Images are tokens, and the token count is the cost model. One token per 28 × 28 block on Qwen2.5-VL after a resize you control; a page is a few hundred to over eleven thousand tokens depending on one parameter. Everything downstream — small-print failures, KV cache, GPU count, the bill — follows from that number.
The model is one stage. Preprocessing decides what it sees, a nullable schema with evidence quotes decides what it can say, a validation ladder decides what to believe, and a review loop turns the rest into labels. “Which VLM” matters less than whether those stages exist.
Measure on your own documents. Public benchmarks are saturated and contaminated; a stratified, double-annotated golden set of 200–500 pages, held out by source, is the first deliverable of any pilot and the regression suite for every model, prompt and engine change after it.
Related folders
- Transformers and LLMs — the language model underneath, the KV cache, the API-side multimodal note.
- Inference and serving — vLLM, batching, quantisation in general.
- Training and fine-tuning — LoRA, PEFT, quantisation methods.
- Computer vision — the classical CV the hybrid pipeline uses.
- Structured Document Pipelines — Parsing, Tables, OCR, Extraction — parsing for retrieval rather than extraction.