AI & ML / Context engineering / 01_context_management.md

Context engineering

Updated 7 interview angles 8 min read source
On this page10
  1. The budget
  2. The mechanism: why position decides cost
  3. Ordering
  4. The compaction that costs more than it saves
  5. Managing history
  6. Memory beyond the window
  7. Where it breaks in production
  8. Sub-agents as context isolation
  9. Related
  10. Interview angle

Context engineering

Article.

Prompt engineering is how you word the instruction. Context engineering is deciding what occupies the window, in what order, and at what cost — and in any real application it is by far the larger lever.

Almost every rule below follows from one mechanism, so it is worth starting there rather than memorising the rules separately.

The budget

Everything shares one window: system prompt, tool definitions, retrieved documents, conversation history, and the response being generated.

python 
budget = (
    context_limit
    - max_output_tokens        # reserved, not available for input
    - n_tokens(system)
    - n_tokens(tool_schemas)   # the line everyone forgets
)
# whatever remains is split between history and retrieval

Tool definitions are the forgotten cost. Twenty tools with detailed schemas and descriptions can run to several thousand tokens, and they are re-sent on every single call before any content exists. A tool you added for one workflow is charged to every request in the application.

The mechanism: why position decides cost

Attention computes a key and a value for every token, and those depend only on that token and the ones before it. Nothing later can change them.

That single property is what makes prefix caching possible: if two requests share a leading run of identical tokens, the K/V for that run is identical too, so a provider can store it and skip the computation.

text 
request 1:  [system][tools][history A][question 1]
request 2:  [system][tools][history A][question 2]
             \_______________________/
              identical prefix -> cache hit, not recomputed

And it is what makes the cost of editing brutal:

text 
[system][tools][turn 1][turn 2][turn 3][turn 4][turn 5]
                        ^
                 edit here
                        \___________________________/
                  everything after must be recomputed

Change one token at position i and every token after it is invalidated. Not because the cache is naive, but because those keys and values genuinely depend on what came before.

Three of the four rules people recite are consequences of that one fact: put stable content first, do not rewrite history casually, and keep the question last.

Ordering

text 
[system prompt]        stable, cacheable, first
[tool definitions]     stable, cacheable
[retrieved context]    best items at this block's EDGES
[conversation history] oldest is compressible
[current question]     LAST, adjacent to generation

Stable-first is the cache argument above. Question-last is an attention argument: models attend most reliably to the beginning and the end of the context and least reliably to the middle, so the thing you want answered sits where attention is strongest. Same reason the best-ranked retrieved chunks go at the edges of their block rather than buried mid-list — see Context windows.

Less is more

The counterintuitive rule, and the one worth stating out loud in an interview: adding marginal context degrades quality.

Extra chunks dilute attention across more tokens and push genuinely relevant material further into the weak middle. Five well-ranked chunks generally beat fifty. If you are passing everything retrieval returned, that is not context engineering, it is context dumping.

The compaction that costs more than it saves

Here is the arithmetic nobody does before shipping a summariser.

You are 40,000 tokens into a conversation. Summarising the oldest 30,000 into 2,000 looks like an obvious win — 28,000 tokens saved on every future call.

text 
before:  [system 2k][tools 3k][history 40k]
         cached from turn 1

after:   [system 2k][tools 3k][summary 2k][recent 10k]
                              ^ new from here on

The summary sits at position 5,000, so every token after it is a cache miss. The next call re-prefills 12,000 tokens at full price instead of reading 45,000 from cache at a fraction of it. You save on token count and pay on token cost, and whether that nets out depends on how many more turns the conversation has.

Two conclusions follow:

  • Compact rarely and in large batches. Each compaction is one re-prefill, so amortise it over as many subsequent turns as possible. Continuous summarising after every turn is close to the worst possible strategy.
  • Measure both numbers. Token count and token cost move in opposite directions here.

Compaction that does not corrupt the transcript

When you do compact, the rules are:

  1. Keep the first message. It carries the task. Dropping it is how an agent forgets what it was doing and starts answering a different question.
  2. Keep recent turns verbatim. That is the working state.
  3. Summarise decisions and constraints, not prose. “Chose Postgres over DynamoDB because of the join requirement” is worth keeping. A paraphrase of the discussion is not.
  4. Never split a tool-call group. An assistant message with tool calls and its results move together or not at all — see The agent loop.
python 
if n_tokens(messages) > THRESHOLD:
    head, middle, tail = messages[:1], messages[1:-KEEP], messages[-KEEP:]
    messages = head + [summarise(middle)] + align_to_group_boundary(tail)

Managing history

Strategy Loses Cache impact
Sliding window everything older, abruptly breaks at the drop point
Summarise older detail; errors compound breaks at each summary
Hierarchical least breaks on re-summarisation
Externalise to memory conversational nuance good - prefix stays stable

Externalising scores best on cache precisely because it does not rewrite the window; it moves content out and puts a reference in, leaving the prefix intact.

Memory beyond the window

Conversation history is short-term memory. Long-term memory is a separate store the agent reads from and writes to.

Type Holds
Working current task state
Episodic what happened before
Semantic facts about the user or domain
Procedural learned how-to
python 
# Write durable facts, never raw transcripts.
facts = extract_facts(conversation)     # "prefers metric", "on the Pro plan"
memory.upsert(user_id, facts, observed_at=utcnow())

# Read only what this turn needs.
relevant = memory.search(user_id, query, top_k=5)

Two failure modes, and both are curation problems rather than storage problems:

  • Bloat. Storing everything until retrieval returns noise. The store grows, relevance falls, and the agent’s memory makes it worse rather than better.
  • Staleness. A fact that was true in March asserted confidently in August. Timestamp facts, expire them, and prefer re-deriving volatile values over remembering them.

Where it breaks in production

A timestamp in the system prompt. This is the highest-cost one-line bug in the field:

python 
system = f"You are a support agent. Today is {datetime.now()}."

It changes every request, sits at position zero, and therefore invalidates the entire cache for every user on every call. Cache hit rate goes to zero and nobody connects it to a helpful-looking line. If the model needs the date, put it at the end of the prompt, or round it to the day so it is stable for 24 hours.

Verify rather than assume — if cache_read_input_tokens is zero across repeated requests, something early is varying.

Unsorted JSON in a cached block. The same dict serialised twice with different key order is a different prefix. Sort keys anywhere the output lands in a cacheable region.

Tool list churn. Adding or removing a tool mid-conversation changes the tool block, which sits early, which invalidates everything. If tools must vary, append rather than swap, or accept the cost deliberately.

Sub-agents as context isolation

The cleanest way to keep a context small is not to compress it — it is to never put the content there.

Delegate a subtask to a fresh context and return only its conclusion. The parent never sees the sub-agent’s twenty tool calls, just the answer. That is the most defensible reason to use multiple agents at all, and it is a context argument rather than an intelligence one — see Multi-agent patterns.

The trade not taken

With a million-token window, the obvious question is whether any of this is still necessary. Sometimes it genuinely is not: if the corpus fits, putting everything in the window beats building retrieval, and it removes an entire subsystem.

What does not go away is that attention degrades over distance and cost scales with tokens. A million-token prompt is expensive on every call, slow to prefill, and reliably worse at using material buried in its middle than a well-built 8,000-token one. Long context changes when you need retrieval; it does not change the fact that what you include is a decision.

Interview angle 7

  • “Context engineering versus prompt engineering?” — prompt engineering is wording; context engineering is what occupies the window, in what order, at what cost. In a real application it is the bigger lever.
  • “How do you order the window, and why?” — stable content first so it caches, the question last so it sits where attention is strongest, and the best retrieved chunks at the extremes of their block rather than the middle.
    • Follow-up: “Why does stable-first matter mechanically?” — a token’s key and value depend only on itself and what precedes it, so an identical prefix produces an identical cache. Anything variable placed early destroys the shared prefix for everything after it.
  • “Should you retrieve more chunks to be safe?” — no. Marginal context dilutes attention and pushes relevant material into the weak middle. Five well-ranked chunks beat fifty.
  • “You summarise history to save tokens. What does it cost?” — a full re-prefill of everything after the edit, because those keys and values genuinely depend on what changed. A compaction that saves 28,000 tokens of count can cost more in price than it saves.
    • Follow-up: “So when do you compact?” — rarely, in large batches, so one re-prefill amortises over many turns. Compacting every turn is close to the worst strategy available.
  • “Your cache hit rate is zero. Where do you look first?” — anything varying early. A datetime.now() in the system prompt is the classic: it looks helpful, sits at position zero, and invalidates every request for every user. Unsorted JSON and a changing tool list do the same.
  • “An agent runs out of context mid-task.” — keep the first message and the recent turns, summarise the middle into decisions and constraints, externalise large artefacts, and never split an assistant tool-call message from its results or the request becomes invalid.
  • “Does a million-token window make this obsolete?” — it changes when you need retrieval, not whether inclusion is a decision. Attention still degrades over distance and cost still scales with tokens, so a well-built small context beats a large one padded to feel safe.