Fine-tuning VLMs

LoRA on a VLM is LoRA on its language model, with two extra decisions the text case does not have — whether to touch the vision side, and how to keep the image preprocessing identical between training and serving. The base mechanics are in LoRA and parameter-efficient fine-tuning; this note is what changes when the input is a page.

When it is the right tool

Fine-tuning a document VLM pays for four things, and the review loop tells you which you have (Structured extraction from documents):

• Your templates and your fields. The base model reads invoices; yours have a “cost centre” box in the corner that it keeps missing. A few hundred corrected examples fix this reliably.
• Format compliance without a long prompt. Baking the schema and the conventions into the weights shortens every request by a thousand tokens and removes a class of format drift.
• A small model matching a large one. Label with the 72B (or a hosted frontier model, if its terms allow), train the 8B on the labels, serve the 8B. The usual reason a per-page cost drops by 5x.
• A visual domain the base has not seen — technical drawings, unusual scripts, dense handwriting. This is the case that needs the vision side unfrozen.

It does not pay for rotation, blur or resolution problems (fix them in preprocessing), for fewer than a couple of hundred examples (prompt and few-shot instead), or for knowledge the model lacks (retrieval).

What to adapt

from peft import LoraConfig

# Qwen2.5-VL: the ViT also has gate_proj/up_proj/down_proj, so naming the
# modules alone would adapt the vision tower by accident. Anchor the regex
# to the language model; print(model) first - names move between versions.
lora = LoraConfig(
    r=32, lora_alpha=64, lora_dropout=0.05,
    target_modules=r".*language_model.*\.(q_proj|k_proj|v_proj|o_proj"
                   r"|gate_proj|up_proj|down_proj)$",
    task_type="CAUSAL_LM",
)

The projector is a few million parameters and often worth unfreezing; the encoder is where forgetting lives — unfreeze it for drawings or a script the base cannot read, with a lower learning rate, and check that the model still reads ordinary print afterwards.

QLoRA puts the 32B and 72B on one 80 GB card: 4-bit frozen base, bf16 adapters, gradient checkpointing, and the pixel budget lowered so a page plus its answer fits the sequence length you can afford.

The data

{"messages": [
  {"role": "system", "content": "<the production system prompt>"},
  {"role": "user", "content": [
     {"type": "image", "image": "pages/inv_0412_p1.png"},
     {"type": "text", "text": "Extract the fields. Use null for ..."}]},
  {"role": "assistant", "content": "{\"invoice_number\": {\"value\": ..."}
]}

Rules that decide whether the run is useful:

• Same prompt, same pixel budget as production. Train at max_pixels 1,003,520 and serve at the default 12.8 million and you have trained on images the model never sees. Resolution skew between training and serving is the VLM-specific bug.
• Targets are the validated output, including nulls and quotes, in the exact JSON layout the grammar will enforce. Teach the model the behaviour you validate.
• Split by document source, not at random. Two pages of one invoice, or two invoices from one vendor template, on both sides of the split is leakage, and the eval will lie.
• 500 to 5,000 labelled pages is the working range for a template-level fine-tune; mix in 5–10% general instruction data to slow forgetting.
• Augment by rendering, not by copying: rotate a few degrees, add scan noise, vary fonts on synthetic pages — the same domain-gap logic as Synthetic data and image normalization.

Frameworks and the run

LLaMA-Factory and ms-swift are the batteries-included choices and support the current Qwen, InternVL, Llama Vision and Pixtral checkpoints with a config file and flags for freezing the vision tower and projector. TRL’s SFTTrainer with a collator that runs the model’s processor is the transparent option; Unsloth is the fast one for the models it supports.

Whichever runs the training, the settings that matter are the same:

• LoRA r 16–64 with alpha = 2r; learning rate ~1e-4 for the adapters, an order lower if the encoder is unfrozen; 1–3 epochs, cosine schedule, short warm-up.
• bf16, gradient checkpointing, flash attention, and a sequence length that fits the page’s image tokens plus the answer — check the longest example before the run, not after it OOMs.
• Evaluate every few hundred steps on the golden set with the production metric (field accuracy), not loss. Loss keeps falling after the useful training has stopped.

Shipping it

Merge the adapter into the base weights and serve the merged checkpoint. Adapter hot-swapping in the serving engine is mature for text models and partial for VLMs, and a merged model is what you quantise anyway: merge, then quantise (AWQ or FP8) with calibration data drawn from your pages, then run the golden set on the quantised artifact, because that — not the bf16 training checkpoint — is what production runs.

Register it with the data hash, the base revision and the eval numbers (Experiment tracking and the model registry).

Where it breaks in production

• Resolution skew. Trained at one max_pixels, served at another. Accuracy drops for no visible reason; the fix is one number.
• The vendor in both splits. A 98% eval that is 90% in production, because the split was random and the templates leaked.
• Forgetting the general case. Perfect on your forms, worse on the odd attachment types it used to handle. Mix general data and test a general subset.
• Quantising before evaluating. The bf16 model passed; the AWQ merge shipped without a run. Evaluate the artifact you serve.
• Training on unvalidated labels. Reviewer corrections that were never double-checked teach the model the reviewers’ mistakes at scale.

Related

• LoRA and parameter-efficient fine-tuning - the mechanics.
• Distillation and synthetic data - labelling with a bigger model.
• Fine-tuning versus RAG - the decision before this one.
• Evaluating document extraction - the metric the run is judged by.