Sizing GPUs against an SLA and a budget

“How many GPUs, and what will it cost” is answered with arithmetic on three measured numbers, not with a vendor’s tokens-per-second chart. The interview version wants the method and one worked example with the assumptions stated; the presales version wants the same thing with a range and a pilot attached. All numbers below are illustrative — the point is that you measure them on your model, your pages and your GPU.

Three numbers, and where they come from

1. Tokens per page: image tokens at the chosen pixel budget, prompt tokens (mostly cached), output tokens. From the processor and a sample of real documents (How a VLM sees an image).
2. Throughput per GPU: prefill tokens per second and aggregate decode tokens per second at the batch size you will run. From a benchmark that replays real pages against the real server — never from a chat benchmark, because the workload shape is different.
3. The SLA: pages per hour and the latency bound, with the peak-to- average ratio. From the client, in writing.

# Replay real pages; the numbers you want are prefill and decode tokens/s
# at the concurrency you intend, and the p95 latency at that concurrency.
vllm bench serve --model qwen3-vl-8b --backend openai-chat \
  --dataset-name custom --dataset-path pages.jsonl \
  --max-concurrency 32 --num-prompts 512

Then the arithmetic:

pages_per_day, peak_ratio = 300_000, 3
img, prompt_uncached, out = 1300, 150, 400          # tokens per page
prefill_tps, decode_tps = 25_000, 3_000             # measured, per GPU

gpu_s_per_page = (img + prompt_uncached) / prefill_tps + out / decode_tps
peak_pages_s = pages_per_day / 86_400 * peak_ratio  # 10.4 pages/s
gpus = peak_pages_s * gpu_s_per_page / 0.7          # 70% target utilisation
print(round(gpu_s_per_page, 3), round(gpus, 2))     # 0.191 s -> 2.85 GPUs

Three H100s, plus one for failure — or two plus a queue, if the SLA lets work wait. The SLA decides which. A “results within 15 minutes” SLA lets a queue absorb the peak and three GPUs suffice; a “p95 under five seconds per page” SLA needs capacity for the peak rate and forces a look at single-request latency, below.

Memory sets the ceiling on concurrency

Throughput at a batch size you cannot fit is fiction. KV cache per page times pages in flight must fit what is left after the weights (Serving VLMs): an 8B in FP8 on an 80 GB card holds hundreds of pages in flight; on a 24 GB card, a dozen.

Below the knee, more concurrency is nearly free throughput; above it, requests queue inside the server and latency climbs while the benchmark still reports a high tokens-per-second. Find the knee, run below it.

Latency is a different budget

A batch SLA is about throughput; an online SLA is about one request. For one page, single-stream:

• Encoder plus prefill of ~1,500 tokens: 100–300 ms.
• Decode of 400 tokens at a single-stream 80–120 tokens per second on an 8B: 3–5 seconds.

Decode dominates, and it is memory-bandwidth-bound, so the levers are the ones that move bytes: FP8 or AWQ weights (fewer bytes per token), shorter output — a compact schema without quotes for low-risk fields halves it — speculative decoding, which helps exactly here at low concurrency, or a smaller model. A p95 of five seconds is not met by adding GPUs; it is met by changing the request.

Cost, and the comparison the client actually wants

At 300,000 pages a day self-hosting is several times cheaper than a hosted API before the engineering and on-call cost, which is real and is the honest line item. At 30,000 pages a day the arithmetic flips: one GPU sitting at 15% utilisation costs about what the API would, and the API comes with no pager.

The break-even is usually somewhere between ten and fifty thousand pages a day, and saying that number for the client’s volume is the whole value of the exercise.

Two refinements that change the answer by 2x:

• Quantisation (FP8 on Hopper/Ada, AWQ elsewhere) raises throughput and concurrency on the same card for a quality cost you measure on the golden set — usually the first lever, not the last.
• Batch pools on spot or reserved capacity for the nightly volume, a small always-on pool for the online path. The peak-to-average ratio is what you are paying for; smoothing it with a queue is cheaper than buying it.

Saying it to a client

The shape of a defensible estimate: a range with the assumptions visible and a pilot that narrows it.

“At your stated 300k pages a day, three to four H100-class GPUs, roughly $0.50–0.80 per thousand pages in compute, assuming ~1,300 image tokens per page and a 15-minute turnaround. A two-week pilot on 5,000 of your documents replaces the assumptions with measurements — tokens per page, field accuracy and the review rate — and we re-quote on those.”

The pilot also produces the golden set (Evaluating document extraction), so it is not overhead; it is the first deliverable.

What sinks estimates: quoting from a benchmark table, forgetting the review team’s time (which usually exceeds the GPU bill), assuming the average rate instead of the peak, and sizing for the model you demoed rather than the quantised one you will ship.

Where it breaks in production

• Capacity from average load. The 09:00 upload spike is 3–5x average; the queue backs up, the SLA is missed exactly when the client is watching.
• Benchmarked chat, deployed documents. Prefill-heavy pages behave nothing like a chat benchmark; throughput is half what the sheet said.
• No headroom. One GPU fails, or a deploy takes one out, and the remaining ones are over the knee.
• Latency SLA promised on throughput numbers. Decode is per-request and bandwidth-bound; batch size does not shorten it.
• Costs quoted without reviewers. The compute is $0.70 per thousand pages; the human review at a 10% review rate is many times that.

Related

• Serving VLMs - the flags and the memory budget.
• KV cache - the per-token arithmetic.
• LLM inference fundamentals - prefill versus decode, the metrics.
• Cut LLM spend without losing quality - the cost levers on the API side.
• Presales and scoping AI work - the conversation around the number.