Why it matters
Long documents, transcripts and tool outputs exceed the budget even after selection. Compression reduces tokens so more sources fit, or so a long conversation can continue. The engineering problem is that compression is a lossy transformation applied before you know which detail matters. A summary that reads well can drop the identifier, date or exception that the question turns on, and nothing in the pipeline notices.
Mental model
Compression is packing a suitcase by vacuum-sealing: same trip, less space, some items deformed. The comparison breaks because you cannot tell which item was crushed until you need it at the destination, and the suitcase gives no receipt of what it removed.
How it works
Context window · illustrative tokens
1,440 input+ 700 reserve= 2,140 of 3,000
860 free
Step 1Fit the window
The summary leaves 860 tokens free, so nothing else has to be dropped to send the request. Flip the History switch to see the transcript it replaced.
Output reserve
700Pick a section to read its token count, its role and its state.
Figure. Compression buys window space by rewriting the transcript. Here 1,600 tokens of history become a 260-token summary that fits, while three named details do not survive the rewrite.
What to notice. The summary text is authored for this example; no model was run and no compression ratio was measured. The point is the shape of the trade — 860 tokens of headroom against three details that are gone — not a benchmark.
Text version
A 3,000-token window with a 700-token output reserve. The default first step has history summarised: instructions 450, question 130, retrieved extract 600 and a 260-token summary, for 1,440 input tokens and 860 free. The switch labelled History toggles between Summary and Full transcript; with Full transcript selected the same window holds four history turns of 350, 300, 700 and 250 tokens and runs 480 tokens over budget, with a hatched band over the reserve. The second step points at what the summary keeps: the order id, the date and the promise to refund to the original card. The third step lists the details it lost — a 500 error code, the first workaround and a stated deadline — as dropped entries, not as separate budget lines. The fourth step is an audit reminder to keep a pointer to the source turns.
- Select first, then compress only what still exceeds the budget.
- Choose the method: extractive spans for auditable evidence, abstractive summaries for background, learned compression only with evaluation.
- Preserve identifiers, numbers, negations and exceptions verbatim, since these are what summaries drop.
- Keep a pointer from each compressed block to its source, so the original can be fetched when needed.
- Measure task accuracy with and without compression on the same cases, per document type.
- Set a compression ratio ceiling, and fall back to fewer sources when the ratio would be too aggressive.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Extractive compression | Faithful and auditable | Less reduction, awkward fragments | The budget is very tight |
| Abstractive summary | Large reduction, readable | Drops detail and can distort | The task needs exact values |
| Learned compression | Strong ratio with training | Model-specific, hard to inspect | You need traceability |
| No compression | Full fidelity | Fewer sources fit | Documents are long and numerous |
Good for
Faithful and auditable
Cost
Less reduction, awkward fragments
Fails when
The budget is very tight
- The budget is very tight
- The task needs exact values
- You need traceability
- Documents are long and numerous
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Extractive compression is good for Faithful and auditable, costs Less reduction, awkward fragments, and fails when The budget is very tight. Abstractive summary is good for Large reduction, readable, costs Drops detail and can distort, and fails when The task needs exact values. Learned compression is good for Strong ratio with training, costs Model-specific, hard to inspect, and fails when You need traceability. No compression is good for Full fidelity, costs Fewer sources fit, and fails when Documents are long and numerous
Failure modes
- Dropped exception. The summary omits the one clause that changes the answer. Debug: compare answers against the original document for cases where compression was applied.
- Distorted number. A rounded or rephrased figure enters the context as fact. Debug: extract numbers from compressed text and check each against the source.
- Compression cascade. A summary of a summary loses more each turn. Debug: track compression depth per session and cap it.
In production
Log the compression ratio and the source ids per request, and audit a sample against the originals. Alert when compression depth rises in long sessions.
Common misconceptions
People think: Compression is a safe way to fit more context.
Actually: It removes information before you know what matters, and the loss is invisible at read time.
People think: A good summary preserves what matters.
Actually: It preserves what the summariser judged important, which may not match the question.
Interview angle
Separate compression from selection, then name what must be preserved verbatim and how you would detect a loss. Strong answers mention source pointers and per-type evaluation. Weak answers summarise everything and assume a smaller prompt is a better one.
Sources
- Jiang et al., LLMLingua — token-level prompt compression and its trade-offs.
- Mu et al., Learning to Compress Prompts with Gist Tokens — compressing prompts into learned tokens.
- Jiang et al., LongLLMLingua — compression for long-context question answering.