Serving VLMs

Serving a vision-language model is serving a language model plus three things that are easy to get wrong: image tokens that blow past the context you sized for, CPU preprocessing that becomes the bottleneck before the GPU does, and a prefix cache that only helps if you let it.

The general engine material is in Batching and serving engines; this is the multimodal delta.

The server

docker run --gpus all --ipc=host -p 8000:8000 \
  -v /models/hf:/root/.cache/huggingface \
  vllm/vllm-openai:v0.11.0 \
  --model Qwen/Qwen3-VL-8B-Instruct --revision <commit-sha> \
  --max-model-len 16384 \
  --limit-mm-per-prompt '{"image": 4}' \
  --mm-processor-kwargs '{"max_pixels": 1003520}' \
  --max-num-seqs 64 --gpu-memory-utilization 0.92 \
  --enable-prefix-caching

Each flag is a decision:

• --revision pins the checkpoint. A repo updated in place is a silent model change.
• --max-model-len must hold images plus prompt plus output: four pages at 1,280 tokens, a 1,000-token system prompt and 2,000 tokens of JSON is 8k, so 16k with margin. It bounds per-sequence memory; do not leave it at the model’s 256k maximum.
• --limit-mm-per-prompt caps images per request. A client sending a 40-page document as 40 images gets a clean error instead of a 50k-token prefill that stalls the batch.
• --mm-processor-kwargs sets the pixel budget server-side, so no caller can send a 12,000-token page by omitting a parameter. This is the single most important VLM-specific flag (How a VLM sees an image).
• --max-num-seqs is the concurrency ceiling; with pages at ~1,300 tokens it is what the KV budget allows (below), not the default.

SGLang serves the same models with the same OpenAI-compatible surface and RadixAttention prefix sharing; TensorRT-LLM behind Triton is the compile-everything option when the model is fixed and the GPU-hours justify the engineering. Triton also earns a place when the pipeline mixes a VLM with classical CV models — YOLO, a layout detector — that it serves from ONNX or TensorRT next to the LLM.

Memory, and therefore concurrency

Weights first, then the KV cache gets the rest. For Qwen3-VL-8B in bf16: about 16 GB of language model plus under a gigabyte of vision encoder. On an 80 GB card at 0.92 utilisation that leaves roughly 55 GB for KV cache; on a 24 GB L4 or A10G, about 5 GB — which is the difference between hundreds of pages in flight and a handful.

def kv_bytes_per_token(layers, kv_heads, head_dim, dtype_bytes=2):
    return 2 * layers * kv_heads * head_dim * dtype_bytes     # K and V

qwen3_8b = kv_bytes_per_token(36, 8, 128)            # ~147 KB / token
page = 1300 * qwen3_8b                               # ~190 MB per page
print(55e9 / page)                                   # ~290 pages in flight
print(55e9 / (page / 2))                             # ~580 with FP8 KV cache

Two levers move this a lot. FP8 KV cache halves the per-token cost for negligible quality loss. Weight quantisation — AWQ or GPTQ at 4 bits, or FP8 on Hopper and Ada — shrinks the 16 GB to 5–9 GB and hands the rest to the cache; the vision encoder stays in bf16 because it is small and sensitive. Qwen publishes AWQ and FP8 checkpoints for exactly this. See Quantization.

Tensor parallelism is for a model that does not fit one card (the 32B in bf16, the 72B, the MoE). For a model that fits, replicas beat TP: two independent 8B servers on two GPUs outperform one TP=2 server, because there is no all-reduce on every layer.

Prefill-heavy, and what that changes

A document request is thousands of image tokens in and a few hundred JSON tokens out. Compared with chat:

• Throughput is prefill throughput. --max-num-batched-tokens — the chunked-prefill budget per step — is the knob; raise it for batch jobs where latency of any single page does not matter.
• Prefix caching pays only for the shared prefix. The system prompt and schema description cache across every request; the image never does unless it is byte-identical (the engine hashes image content, so a duplicated page is cached, a re-scanned one is not). Put the stable text before the image in the message.
• Speculative decoding barely helps. It accelerates decode; there is little decode here.
• The vision encoder is a separate cost with its own cache; a request with four images runs it four times before prefill starts.

The CPU is the other bottleneck

Decoding the PNG, resizing to the pixel budget and normalising happen in the API server process on CPU, and Qwen’s processor at high resolutions is not fast. At tens of pages per second the GPU idles waiting for the preprocessor.

Fixes, in order: pre-resize in the worker to exactly the target dimensions so the server’s resize is a no-op; send JPEG at the final size rather than a 5 MB PNG in base64; run more API-server processes in front of the engine; and watch CPU on the serving pod, not just GPU.

Send bytes, not URLs. A server that fetches image_url from arbitrary hosts is an SSRF endpoint; restrict allowed media domains or disable fetching and pass base64.

Batch versus online

The same server does both, but the shapes differ:

For nightly volume, the offline path — vllm run-batch on a JSONL file, or the Python LLM.generate API in a worker — removes the HTTP layer and the per-request overhead entirely. Mixing a batch flood into the online server is how p95 explodes at 02:00; separate the pools or apply priorities.

Operating it

• Readiness is minutes, not seconds. Model load and CUDA graph capture take one to several minutes; a readiness probe that fires early routes traffic into 503s. Weights live on a volume or a pre-baked image, never downloaded per pod start.
• /metrics is Prometheus. num_requests_waiting, KV cache usage, TTFT and per-token latency histograms. Autoscale on queue depth, not on GPU utilisation, which sits near 100% whether or not the server is keeping up.
• A serving upgrade is a model change. New vLLM, new default preprocessing or kernels; the golden set runs before the rollout.
• Timeouts at every hop. A looping generation should die at max_tokens, and the client’s timeout must exceed the worst legitimate page, or workers retry work that was about to finish.

Related

• Sizing GPUs against an SLA and a budget - turning the numbers above into a GPU count and a cost.
• Constrained decoding on self-hosted models - the structured-output backend inside this server.
• KV cache - the arithmetic in full.
• Model export and inference runtimes - Triton and TensorRT.
• Docker and Docker Compose Command Guide - the container side.