KV cache

Article.

Verified 2026-08

Verified 2026-08.

A GPU capable of roughly a quadrillion floating-point operations per second produces about fifty tokens per second from a large model. Those numbers are five orders of magnitude apart, and the gap is not inefficiency — it is the correct answer for what generation actually does.

Generation is memory-bound, not compute-bound. That single fact explains the KV cache, why batching helps so much, why output tokens cost more than input tokens, why long contexts collapse throughput, and why speculative decoding works at all.

Why the cache exists

Generation is autoregressive: produce a token, append it, run the model again. Without caching, generating token 1000 recomputes attention over all 999 previous tokens — the entire sequence, every step.

But keys and values for previous tokens never change. Cache them, and each step computes only its own K and V:

def decode_step(x_new, cache):
    q, k, v = project(x_new)                  # only the new token
    cache.k = torch.cat([cache.k, k], dim=-2)
    cache.v = torch.cat([cache.v, v], dim=-2)
    return attention(q, cache.k, cache.v)     # attend over everything so far

Per-token model cost drops from O(n) to O(1). The price is holding the cache in memory — and that price turns out to dominate everything else.

The mechanism: arithmetic intensity

The number that decides whether a workload is compute-bound or memory-bound is arithmetic intensity: FLOPs performed per byte read from memory.

Decoding one token at batch size 1 does roughly 2 FLOPs per parameter and must read every parameter — 2 bytes each in fp16.

intensity = 2 FLOPs / 2 bytes ≈ 1 FLOP per byte

A modern accelerator needs somewhere around 100+ FLOPs per byte before its compute units are the constraint. At intensity 1 you are roughly two orders of magnitude below the ridge point: the GPU spends essentially all its time waiting for weights to arrive.

The ceiling is calculable and it is not approximate:

70B model, FP8 weights          = 70 GB read per token
H100 HBM bandwidth              ≈ 3.35 TB/s
--------------------------------------------------
70 / 3350  ≈ 21 ms per token    ≈ 48 tokens/second

That is a hard bound from memory bandwidth alone, with a hypothetically infinite-FLOPs GPU. No kernel optimisation moves it, which is why single- stream generation speed tracks memory bandwidth across hardware generations far more closely than it tracks FLOPs.

Which is why batching is the whole game

Batching does not read the weights once per request. It reads them once per batch, and each additional sequence adds FLOPs without adding weight traffic:

batch 1    intensity ≈ 1     hopelessly memory-bound
batch 32   intensity ≈ 32    much better
batch 128  intensity ≈ 128   approaching compute-bound

Arithmetic intensity is approximately the batch size. That is the entire economic argument for a serving engine: throughput per GPU scales close to linearly with batch size until you hit the ridge, and what stops you reaching it is KV cache memory, not compute.

So the chain is: decode is bandwidth-bound → batching fixes it → the KV cache caps the batch → cache size is the thing that determines your cost per token.

Prefill and decode are different workloads

Prefill has thousands of positions to work on simultaneously, so its arithmetic intensity is high and it saturates the GPU. Decode has one.

This is why providers price input tokens below output tokens — and being able to explain the pricing from the hardware is a strong signal, because most candidates treat it as an arbitrary business decision.

Sizing it

bytes = 2 * layers * kv_heads * d_head * seq_len
            * batch * dtype_bytes

The 2 is K and V. Worked example — a 70B-class model, GQA with 8 KV heads, d_head 128, 80 layers, fp16, 32k context, one request:

2 * 80 * 8 * 128 * 32768 * 2 bytes  ≈  10.7 GB

Eight concurrent requests exceed an 80 GB GPU before the weights are loaded. With plain multi-head attention at 64 KV heads it would be eight times that.

That calculation is the entire reason GQA and MLA exist — see Attention variants. They are not quality improvements; they are memory-footprint reductions that buy batch size, and batch size is throughput.

Shrinking it

FP8 KV cache is the free win and is standard in 2026 serving stacks. Halving cache memory roughly doubles concurrency at essentially no quality cost, and it is one flag.

What the serving layer does with it

Three techniques all exist because of the numbers above, and they are covered in Batching and serving engines:

• PagedAttention — manage the cache in fixed-size blocks like OS virtual memory, removing fragmentation and over-allocation, and allowing blocks to be shared between sequences.
• Prefix caching — that sharing applied to a common system prompt, so the shared prefix is prefilled once for all requests.
• Chunked prefill — interleave prefill work with decode steps so a long prompt does not stall everyone else’s generation.

The one design consequence worth carrying into a prompt-design discussion: put stable content first and variable content last. A prompt shaped as [system][tools][examples][user query] shares a long prefix across every request; injecting anything user-specific early destroys it.

Where it breaks in production

A timestamp in the system prompt

The single most expensive one-line change available:

You are a helpful assistant.
Current time: 2026-08-20T14:33:07Z   <- every request misses

The prefix diverges at token ~12, so nothing after it is reusable, and a workload that had 95% prefix cache hits has zero. Latency and cost roughly double and the diff looks harmless.

If the model genuinely needs the time, put it at the end, next to the user message, or round it to the hour so it is stable across requests.

Tool definitions that reorder

The same failure with a less obvious cause: tool schemas serialized from a dict, or a set, or with keys in whatever order the ORM returned them. The bytes differ between requests even though the content is identical, so the prefix hash misses.

Serialize tool definitions deterministically — sorted keys, fixed ordering — and treat the serialized prompt prefix as a build artefact rather than something constructed ad hoc per request.

Long context collapsing throughput

Doubling context does not double cost — it can quarter throughput. The cache grows linearly per request, which shrinks the batch that fits, and throughput scales with batch size, so the two effects compound.

 4k context  ->  60 concurrent  ->  high throughput
32k context  ->   7 concurrent  ->  throughput collapses

This is why “just use the million-token window” is a cost decision rather than a capability decision. See Context windows.

Preemption that looks like random latency

When the cache fills, the engine must free space, so it preempts a running request — swapping or recomputing its cache later. The user sees a generation that stalls mid-stream for no visible reason, and it correlates with nothing in your application logs.

Watch the engine’s preemption counter, not just p99 latency. Rising preemptions mean the cache is the binding constraint and the fix is FP8, a lower --gpu-memory-utilization ceiling for headroom, or fewer concurrent long contexts — not a bigger machine.

Summarising the history and losing the cache

A conversation-management strategy that rewrites earlier turns is trading a shorter context for a cache miss on everything after the rewrite. The next turn re-prefills the whole compacted history.

Both sides are real: compaction reduces per-token cost forever, the rewrite costs one full prefill. Measure both. Appending is cache-friendly; editing is not, so a strategy that appends a summary and drops old turns below the stable prefix behaves very differently from one that rewrites the prefix itself. See Context engineering.

Speculative decoding, and why it works here

Worth understanding as a consequence of the mechanism rather than as a separate trick. If decode at batch 1 is memory-bound, then verifying k candidate tokens costs almost the same as generating one — the weights are read once either way, and the extra FLOPs are free at intensity 1.

So a small draft model proposes k tokens, the large model verifies them in a single pass, and accepted tokens come nearly free. The acceptance rate decides the speedup, and it degrades when the draft model disagrees — which is why it helps most on predictable text and least on the hard tokens you cared about.

The trade not taken

Do not build this. Every number above is a reason to use vLLM, SGLang, TGI or a hosted endpoint rather than a hand-rolled loop: continuous batching, paged memory, prefix sharing and quantised caches are each individually worth more than any application-level optimisation you can make, and together they are the difference between a workable cost model and an unworkable one.

The honest limits of the whole exercise:

A smaller model beats every one of these techniques. Halving the parameter count halves the bytes read per token, which is the actual bottleneck — no cache strategy competes with that. Routing easy requests to a small model is usually the largest single cost lever available, and it requires no serving expertise at all. See Which model for which task.

And sometimes the answer is not to generate. A cached response, a retrieved answer, or a classifier where a generative model was used out of habit costs nothing per token because there are no tokens. The most memory-bound workload is the one you did not run.

Related

• Attention variants - GQA and MLA, which exist for this
• Batching and serving engines - what the engine does about it
• Context windows - what long context costs