AI & ML / Transformers & LLMs / 08_context_windows.md

Context windows

Updated 7 interview angles 10 min read source
On this page9
  1. It is one shared budget
  2. The mechanism: attention is a fixed budget too
  3. Needle-in-a-haystack, and why it misleads
  4. Long context or RAG
  5. Managing the budget
  6. Where it breaks in production
  7. The trade not taken
  8. Related
  9. Interview angle

Context windows

Article.

Verified 2026-08

Verified 2026-08.

The advertised context length is an upper bound on tokens the model will accept, not on tokens it will use well. Those are different numbers, and the gap between them is where the interesting engineering is.

Three separate things limit usable context, they have nothing to do with each other, and confusing them means applying the wrong fix:

  1. Economics — attention is quadratic at prefill and the KV cache is linear.
  2. Attention — a softmax conserves probability mass, so more tokens means less attention each.
  3. Training — most very long windows are rescaled after pretraining at a much shorter length.

It is one shared budget

The window covers the system prompt, tool definitions, history, retrieved documents and the generated response.

python 
available_for_input = context_limit - max_output_tokens - overhead

Fill it with input and there is no room to answer. The failure mode is a truncated or empty response rather than a clean error, which is why this is annoying to diagnose the first time.

The mechanism: attention is a fixed budget too

Everyone knows attention is quadratic in cost. The part that explains the quality behaviour is different and less discussed.

Attention weights come from a softmax over all positions, so they sum to 1. The total is fixed no matter how many tokens there are:

text 
5 relevant chunks       50 chunks, 5 relevant

█████ ← mass on         █▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏
      what matters      ↑ same mass, 10x the tokens

Adding irrelevant context does not merely fail to help — it mathematically takes attention away from the relevant content. That is why “retrieve more, to be safe” is actively wrong past a point, and it is a good thing to say unprompted because the intuition runs the other way.

Lost in the middle

Empirically, retrieval accuracy against position in the context is U-shaped: strong at the beginning, strong at the end, weakest in the middle. A fact placed halfway through a long prompt is measurably more likely to be missed than the same fact at either end.

Part of the reason is attention sinks: models learn to dump surplus attention mass onto the first few tokens, which anchors the start of the sequence. Recency handles the end. The middle has neither.

python 
def fold(chunks):
    """Best-ranked chunks land first and last; weakest get buried."""
    head, tail = [], []
    for i, c in enumerate(chunks):        # already ranked by score
        (head if i % 2 == 0 else tail).append(c)
    return head + tail[::-1]

prompt = render(fold(ranked[:5])) + "\n\n" + question

Three design consequences:

  • Ranking order and prompt order are not the same thing. Rank by score, then fold so the best material sits at both extremes.
  • Put the question after the documents, adjacent to where generation starts.
  • Fewer, better chunks beat more chunks. Retrieving 50 where 5 suffice dilutes attention and pushes relevant content into the weak region — both mechanisms at once.

Extended windows are not uniformly trained

Most very long windows come from rescaling RoPE after pretraining at a much shorter length — position interpolation, NTK-aware scaling, YaRN. See Positional encoding.

The extension works, and quality typically degrades as you approach the advertised maximum. A “1M model” pretrained at 32k and rescaled is not equally capable across the whole range, and no benchmark headline tells you where it stops being reliable for your task.

The practical test is cheap and nobody runs it:

python 
for length in (8_000, 32_000, 128_000, 512_000):
    ctx = pad_with_distractors(real_task_context, target=length)
    scores[length] = evaluate(model, ctx, your_task_suite)
# Where this curve bends is YOUR context limit, not the advertised one.

Needle-in-a-haystack, and why it misleads

Hide a specific string in a long document, ask for it back. Models pass this at impressive lengths, and it is a weak test, because retrieving one verbatim span is far easier than:

  • Aggregating across many positions — “how many times does X appear”
  • Reasoning over dispersed facts — “does section 3 conflict with section 47”
  • Ordering or comparing across distant regions
  • Noticing an absence — “which required clause is missing”

A model scoring 99% on needle retrieval at 128k can fail badly at multi-fact reasoning over the same 128k. The last one is worth naming: negative questions are the hardest shape, because there is no span to attend to.

Evaluate on your actual task shape.

Long context or RAG

Long context RAG
Cost per query grows with corpus roughly flat
Latency grows with prompt flat
Corpus size bounded by window unbounded
Freshness resend everything update the index
Attribution weak strong
Failure mode silently misses retrieves wrong, visibly

That last row decides more real architectures than the cost row. A retrieval failure is visible — you can see what was retrieved and tell it was wrong. A long-context failure is a confident answer that quietly ignored something in the middle, with nothing to inspect.

RAG stays correct for large or changing corpora, for cost control, and wherever citations matter. Long context is better for a single bounded document needing holistic reasoning — a contract, one module, one transcript.

The 2026 default is both: retrieve to narrow the candidate set, then be generous with context over what survives. See RAG and embeddings.

Managing the budget

Strategy What it loses
Truncate oldest turns early context, permanently
Summarise older turns detail; errors compound
Re-retrieve per turn implicit continuity
Hierarchical plus recent least, most complex

Note that summarising or editing history breaks prefix caching from the point of change, so everything after it must be re-prefilled. That is a real cost against the token saving and worth measuring rather than assuming. See Context engineering.

Where it breaks in production

Conversation cost grows quadratically

The one that surprises people on the invoice. Each turn re-sends the entire history, so the tokens billed across a conversation are a triangular number, not a linear one.

text 
turn 1:   1 message   →     500 tokens
turn 2:   2 messages  →   1,000
turn 10: 10 messages  →   5,000
turn 30: 30 messages  →  15,000

total billed over 30 turns ≈ 232,000 input tokens

Prefix caching makes most of that much cheaper, which is exactly why the cache-breaking strategies above deserve measurement — a “token-saving” compaction that halves the context but destroys the cache can cost more than it saves.

No room left for the answer

Input consumed the window, max_tokens had nowhere to go, and the response is truncated mid-sentence. There is usually no error: the API returns a valid response with a length finish reason that nothing in the application checks.

Check the finish reason. A response cut short by the limit is a different outcome from a completed one, and treating them the same means silently shipping half-answers.

Token counting that drifts

len(text) // 4 is a rule of thumb that is wrong in the direction that hurts: code, JSON, non-Latin scripts and long identifiers all tokenise far denser than prose. The estimate says 100k, the real count is 140k, and the request fails in production on the inputs that matter most.

Count with the model’s own tokeniser, and count the fully rendered request — tool schemas, system prompt and message formatting included, not just the user text.

The framework that trims silently

Many chat frameworks drop messages to fit the window, oldest first, and some will happily drop the system prompt or half a tool definition. The model then behaves strangely for reasons invisible in your code.

Own the trimming. Decide explicitly what is droppable and what is not, and log it when something is dropped.

top_k that nobody tuned

The retriever returns 50 chunks because 50 was the example in the tutorial. Attention is diluted across ten times the necessary tokens, the relevant chunk sits at position 31 in the weak middle, and the answer is confidently wrong.

Tune top_k as a real hyperparameter against your eval set. For most tasks the accuracy curve peaks in single digits and declines after.

Evaluating short, deploying long

The eval suite uses 2k-token contexts because those were easy to build. Every degradation described above is invisible until real documents arrive.

Build at least part of the eval set at the context lengths you actually serve — see Building eval sets.

The trade not taken

The cheapest context is the context you did not send.

Every strategy here manages a budget. The alternative is to need less of it: retrieve instead of dumping, give the model a tool to fetch a section on demand instead of pasting the document, or split one large task into several small ones whose contexts are each small. That last option is unfashionable and frequently the best answer, because it also makes each step evaluable.

And one thing worth being blunt about: a context window is not memory. It is a buffer that vanishes when the request ends. A larger window does not give an agent recall across sessions, does not persist anything, and does not substitute for a store you control. Teams reach for a bigger window when what they actually needed was state — see Context engineering.

The reason “just use the million-token window” keeps losing is not that long context does not work. It is that it converts a retrieval problem into a cost problem and an attention problem, and the second one fails silently.

Interview angle 7

  • “What actually limits usable context?” — three independent things: quadratic attention cost at prefill and a linear KV cache, attention dilution because the softmax mass is fixed, and the fact that most long windows were rescaled after pretraining at a shorter length. The advertised number bounds tokens accepted, not tokens used well.
  • “What is ‘lost in the middle’?” — accuracy against position is U-shaped: strong at the start and end, weakest in between. Attention sinks anchor the beginning and recency handles the end; the middle has neither. So order retrieved content toward both extremes and put the question last.
    • Follow-up: “So should you retrieve more chunks to be safe?” — the opposite. Attention is a softmax, so the total mass is fixed; adding irrelevant tokens takes attention away from the relevant ones, and it also pushes them into the weak middle. Fewer, better-ranked chunks generally win, and top_k should be tuned rather than inherited from a tutorial.
  • “Models pass needle-in-a-haystack at long lengths. Is it solved?” — no. Finding one verbatim span is much easier than aggregating, comparing distant regions, or noticing something absent — and absence is the hardest, because there is nothing to attend to. Test on your own task shape at several lengths and find where the curve bends.

Under pressure

  • “Has long context replaced RAG?” — no, and the deciding factor is usually the failure mode rather than the cost. A retrieval failure is visible; a long-context failure is a confident answer that quietly ignored something in the middle. RAG also wins on unbounded corpora, freshness and citations.
    • Follow-up: “So when is long context the right call?” — one bounded document that needs holistic reasoning, where chunking would destroy the relationships you care about. A contract, a transcript, one module. In production the common shape is both: retrieve to narrow, then be generous over what survives.
  • “Your chat app gets slower and more expensive as conversations grow.” — each turn re-sends the full history, so billed tokens grow as a triangular number, and the KV cache grows so each decode step reads more. Prefix caching covers most of it, which is why a compaction strategy that rewrites history needs measuring on both sides.
  • “The response comes back cut off.” — input consumed the window and there was no room for output. It is not an error; the API returns a length finish reason that most applications never check.
  • “What would you do instead of a bigger window?” — send less. Retrieve, or give the model a tool to fetch the section it needs, or split the task into smaller ones that are each easier to evaluate. And note that a window is not memory — it disappears with the request, so a team asking for more context often actually needs persistent state.
Previous
Reasoning models and test-time compute