Guide · Context & Memory

Compaction or truncation — what happens when context runs out?

Published 1 August 2026

When a conversation outgrows the context window, something has to go. There are two mechanisms for getting it out — and they fail in completely different ways. Truncation drops old turns verbatim. Compaction replaces them with a summary. The third option is not an eviction policy at all: don't let the information live in the transcript in the first place. Most people never choose deliberately — they inherit whatever their tool does, then spend an afternoon debugging a model that "suddenly forgot" a decision made an hour earlier.

What each one actually does

Truncation evicts whole messages, oldest first, until the remaining transcript fits. Everything that survives is exact: the same tokens the model saw the first time. Everything that leaves is gone with no trace. The system prompt and any pinned instructions normally survive, because they sit outside the sliding part of the window.

Compaction takes the same old turns and asks a model to write a summary of them, then continues with summary + recent turns. Nothing disappears entirely, but nothing older is exact any more. You have traded fidelity for coverage, and you have introduced a second model call whose output you rarely read.

How to tell which one bit you

The symptoms are distinguishable, and knowing which you are looking at saves hours:

  • Truncation symptom: a clean amnesia boundary. The model has no idea a file was renamed, does not reference it, does not hedge. Ask "what did we decide about X?" and you get a confident answer about something else, or an honest "I don't have that".
  • Compaction symptom: lossy paraphrase. The model remembers that a decision existed but gets the specifics subtly wrong — the right table, the wrong column; the right constraint, the wrong threshold. This is worse to debug precisely because it looks like memory.

A quick diagnostic: ask the model to quote the earlier instruction verbatim. If it cannot quote but can paraphrase, you are on the compacted side of the boundary. If it cannot do either, the turn was truncated away.

Which to choose

The decision follows from one question: is the value of the old context in its exact wording, or in its gist?

  • Exact wording matters → truncate, and pin what must survive. Code review, legal text, log analysis, anything where a paraphrase of a line is a different line. Do not let a summariser stand between the model and a stack trace.
  • Gist is enough → compact. Long design discussions, multi-hour research sessions, customer conversations where the arc matters more than the phrasing.
  • Both matter → neither, on its own. Move the durable part out of the transcript entirely: a file, a scratchpad, a database row, standing instructions. Then it does not compete for window space and does not need to survive a summariser.

The hybrid that actually works

In production the useful configuration is rarely pure. A pattern that holds up:

  1. Pin the invariants. Constraints, conventions, the definition of done — in the system prompt or a standing-context file, never in the rolling transcript.
  2. Externalise decisions as they are made. When something is settled, write it to a durable artefact and say so in the conversation. Now compaction losing the discussion costs nothing.
  3. Compact the middle, keep the tail exact. Summarise everything older than the last N turns; leave the recent window untouched so current work stays verbatim.
  4. Never compact tool output — replace it. Large tool results should be reduced at the moment they are produced, by code you control, not by a summariser later.

A worked example

A four-hour refactor session. Around turn 60 the window fills. Under pure truncation, the "never touch the generated client" rule from turn 3 is evicted and the model edits the generated client on turn 62. Under pure compaction, the rule survives as "avoid editing generated files where possible" — and the model, reading a hedge, edits it anyway with an apology.

Both failures come from the same mistake: a hard rule was left in the rolling transcript. Moved to standing context, the rule survives every eviction policy, and the choice between compaction and truncation becomes a low-stakes preference about how you want the discussion history to degrade.

Cost and latency, briefly

The intuition is that truncation is free — you drop tokens and pay for fewer. That is only true without prompt caching, and most production setups have it on.

Truncation invalidates the cache prefix. A cache hit requires that the beginning of your request is byte-identical to what was cached. Dropping the oldest turns changes the prefix, so the next call re-reads the whole remaining transcript as uncached input — at full input price and full prefill latency. A 90k-token context that would have been served as a cache read can cost several times more, and land noticeably slower, than if you had simply let it sit there. Every subsequent truncation pays it again. Appending to the tail, by contrast, keeps the prefix intact. (Prompt caching gets its own guide in this track; until then, the rule of thumb is: churn the front of the context and you pay for all of it.)

Compaction costs an extra model call, and that call runs at the worst possible moment: mid-task, on a large input, while the user waits. It also rewrites the prefix, so it invalidates the cache too — but it does so once and then leaves a much shorter transcript that re-warms cheaply, where repeated truncation invalidates over and over.

Practical consequences: don't evict on every turn — evict in large steps so you pay the cache miss rarely instead of constantly. Keep everything stable at the front (system prompt, standing context, tool definitions) and let churn happen at the tail. If your tooling compacts automatically, expect a visible stall at the boundary; compacting on a schedule rather than on overflow moves that cost to a moment nobody is watching.

Review checklist

  • Do you know which mechanism your tool uses, or are you assuming?
  • Is every hard rule outside the rolling transcript?
  • Are decisions written somewhere durable at the moment they are made?
  • Is large tool output reduced by your code before it enters the window?
  • Can you name the last thing the model lost, and when?
  • Do you know what your eviction policy does to the prompt-cache prefix — and how often it fires?

Where this fits

This is a context-budgeting decision, so it sits in the Context & Memory part of the track. The judgement being tested is scope and lifetime: how long a piece of information has to survive, and whether its value is in the exact wording or in the summary. Get that right and the eviction policy stops mattering much.

Related in this track

Keep going