Agent context management: budgets, truncation, compaction, and knowing when to stop

Models now accept hundreds of thousands of tokens of context, and agents fill it fast: tool definitions, file contents, command output, history. Most agent slowdowns and many agent failures I see are context failures, not model failures. Here’s how I manage it.

Every token costs on every step

Context isn’t paid for once. At decode time, every new token reads the whole KV cache of the attention layers. On my box, the bandwidth ceiling drops from about 59 passes a second with a short context to about 47 with a full 262K context (the decode note has the math). Long contexts also make prefill slower, the KV cache bigger and recall less reliable: a fact 200K tokens back is easier to miss than one 2K back.

A context window is a budget, not a bin.

Budget it explicitly

I split the window into fixed allowances and enforce them in the harness:

Part	Typical share	Notes
System prompt and tool schemas	fixed, a few K tokens	Stable, first, never edited: it’s the prefix the cache reuses
Working memory (plan, key facts)	small, bounded	Rewritten deliberately, not appended forever
History	most of the window	Append-only between compactions
Current tool output	capped per call	The part most likely to explode
Reserve for the reply	fixed	Never let a giant tool output eat the room to answer

Tool schemas deserve attention of their own. Fifty tools at about 200 tokens each is 10K tokens on every request. Load only the tools a task needs, or group rarely used ones behind a single “more tools” entry.

Truncate tool output, but keep the signal

Command output is the classic budget breaker: a test run, a log, a find over a big repo. Truncation rules that work:

• Keep the head and the tail. Errors are usually at the end and context at the start.
• Say what was cut, for example [... 4,812 lines omitted ...], so the model knows the gap exists and can ask for more.
• Prefer narrower commands to bigger outputs. Teach the agent to grep and read line ranges instead of reading whole files.
• Summarise structured output structurally. For a test run, report how many passed, how many failed, and the failure messages in full.

Compaction without breaking the cache

Eventually history has to shrink. The naive approach rewrites it at the top of the context every turn, which invalidates the prefix cache and forces a full re-prefill every time (the prefix caching note explains why).

Compact in rare, deliberate steps instead. When history crosses a threshold, replace the oldest span with a summary of decisions, facts and open items, accept one cache miss, and go back to append-only. Ten turns of a reused prefix beat ten turns of an always-changing one.

Isolate work in sub-agents

The cleanest compaction is never to put the noise in the main context at all. Delegate a contained task, like “find where this config is loaded” or “run the tests and report failures”, to a sub-agent with its own fresh context. Only its short result returns to the main agent. The exploration’s thousands of tokens of file contents never touch the parent’s window. amber supports sub-agents for exactly this, and my benchmark has a delegation scenario.

Know when to stop

Loops are context failures too: the agent re-reads the same file, re-runs the same command and appends the same output, until the budget is gone. Detection belongs in the harness:

• Identical-call detection. The same tool with the same arguments twice in a row is almost always a loop. Refuse it, or return a pointer to the previous result.
• No-progress detection. Several steps with no new file changes, no new information and the same error should force a different plan or an escalation.
• Hard limits. Maximum steps, tokens and wall time, with a clean handover to a human when they’re hit.

amber-bench has a whole suite for this, loop-live, and the spread between models on it is wide. Qwen3.6-27B scored 89.0 and the MoE variant 65.9 (the leaderboard has all ten).

Rules I use

1. Budget context per part, and enforce the budgets in code, not in the prompt.
2. Cap and shape every tool output. Head and tail, an explicit omission marker, and narrow commands.
3. Compact rarely and at the end of the prefix. Protect the stable prefix the cache reuses.
4. Delegate exploration to sub-agents. Return results, not transcripts.
5. Detect loops in the harness. Don’t rely on the model to notice.

Sources: the bandwidth figures are from the decode note; the loop-discipline results are from my amber-bench leaderboard. The rest is practice from building amber and client harnesses.