How a VLM sees an image

Every open-weight vision-language model is a language model with an image path bolted on, and the shape of that path decides what the model is good at, what a page costs, and why it fails on small print. Interviewers who run document workloads ask about this because the token arithmetic is the cost model.

Three ways to attach eyes to a language model

encoder-decoder (LLaVA lineage)
  image ─▶ ViT ─▶ projector ─▶ [img tokens][text] ─▶ LLM

cross-attention (Flamingo lineage)
  image ─▶ ViT ─▶ K,V ──┐
  text tokens ─▶ LLM ───┴─ gated cross-attn layers

early fusion
  image patches and text share one token stream from
  pretraining onward

Encoder–projector–decoder is the dominant design: Qwen-VL, InternVL, Pixtral, Gemma 3, MiniCPM-V, LLaVA. A vision transformer turns patches into vectors, a small MLP projects them into the LLM’s embedding space, and they enter the sequence as ordinary tokens. The decoder cannot tell an image token from a word, which is why it reasons across both — and why every image token occupies context and KV cache exactly like text.

Cross-attention (Llama 3.2 Vision) keeps the text model frozen and inserts gated cross-attention layers that read image features. Text-only quality is preserved by construction, image features do not consume the text context window, and the price is a separate path that was trained on less data and grounds worse.

Early fusion (Llama 4, Chameleon-style models) trains on interleaved image and text tokens from the start. Rare among the models you can download and fine-tune; know the name.

Resolution is the whole story

The vision encoder was pretrained at one size — 224, 336, 448 or 560 pixels square. What a model does with a 2,480 × 3,508 A4 scan is the design decision that separates the families:

Fixed means the page is squashed to the encoder’s size; 8-point text is gone before the model sees it. Tiling cuts the page into encoder-sized crops and adds a downscaled overview, so detail survives at the cost of N times the tokens and seams between tiles. Native dynamic resolution runs the encoder on the page at (roughly) its own size with 2D positional encodings, producing a token count proportional to the pixel count.

The token arithmetic

Qwen2.5-VL is the model to know the numbers for: 14-pixel patches, merged 2 × 2 after the encoder, so one token per 28 × 28 pixel block, after the image is resized to multiples of 28 within a pixel budget:

import math

def smart_resize(h, w, factor=28, min_pixels=256 * 28 * 28,
                 max_pixels=1280 * 28 * 28):
    """Qwen2-VL's resize: multiples of `factor`, area inside the budget."""
    hb = max(factor, round(h / factor) * factor)
    wb = max(factor, round(w / factor) * factor)
    if hb * wb > max_pixels:
        beta = math.sqrt(h * w / max_pixels)
        hb = math.floor(h / beta / factor) * factor
        wb = math.floor(w / beta / factor) * factor
    elif hb * wb < min_pixels:
        beta = math.sqrt(min_pixels / (h * w))
        hb = math.ceil(h * beta / factor) * factor
        wb = math.ceil(w * beta / factor) * factor
    return hb, wb

hb, wb = smart_resize(3508, 2480)            # A4 at 300 dpi
print(hb, wb, (hb // 28) * (wb // 28))       # 1176 840 1260

What follows from the arithmetic

Three consequences:

• max_pixels is a quality-versus-cost dial you own. The processor’s default allows about 12.8 million pixels — a 300-dpi A4 scan passes through at 3,500 × 2,492 and costs 11,125 tokens. The model card’s recommended window of 256–1,280 tokens is where most production pipelines sit, and there the same page is rendered at 1,176 × 840: fine for a form, blind to footnotes.
• A page costs more than the prompt around it. Ten pages at 1,280 tokens is 12,800 tokens of prefill before a word of instruction. Image tokens dominate latency, KV cache and therefore concurrency.
• Small text needs pixels, and pixels cost tokens. The fix for unreadable detail is not a bigger max_pixels on every page; it is cropping the region and sending the crop. See VLM failure modes in production.

InternVL’s tiles are the other arithmetic: each 448² tile becomes 256 tokens after pixel-unshuffle, plus one thumbnail tile, so a page at 12 tiles is 3,328 tokens. Llama 3.2 Vision attends to up to four 560² tiles through cross-attention; they cost compute but not context.

Positions and the things a grid loses

The encoder gives each patch a 2D position; the decoder needs to know the image tokens are a grid, not a line. Qwen-VL’s answer is M-RoPE — rotary position split into temporal, height and width components, so a token’s position is a coordinate rather than an index (and video frames get time). Qwen3-VL interleaves the components; InternVL3 uses variable position increments so image tokens do not exhaust the context’s positional range.

What the design does not preserve is fine geometry. Patches are pooled, merged 2 × 2, projected and then read by a model trained to predict words. Holistic understanding survives that; measurement does not. This is the mechanical reason counting, exact bounding boxes and reading a ruler are weak, and why a detector or a classical CV step still wins for those — Object detection.

Where the compute goes

request ─▶ CPU  decode, resize, normalise  (up to 100s ms)
        ─▶ GPU  vision encoder             (once per image)
        ─▶ GPU  prefill: image + text      (dominant here)
        ─▶ GPU  decode: output tokens      (short for JSON)

Document extraction is prefill-heavy and decode-light: thousands of image tokens in, a few hundred JSON tokens out. That inverts the usual chat profile, and it is why batching many pages and caching the shared prompt prefix matter more than speculative decoding for this workload — Serving VLMs.

Related

• Open-weight VLMs: the landscape - which model uses which design.
• Multimodal models - the API-side view of the same mechanism.
• KV cache - why image tokens are a memory problem.
• Positional encoding - RoPE, which M-RoPE extends.