Background
The most frequently encountered engineering problem in Agent development is running out of context window. A 128K or even 200K window looks generous on paper, but tokens are billed by volume; running a 200K context round costs a few cents, and a few hundred rounds per day drives costs up fast. The longer the context, the higher the time-to-first-token latency, and user experience drops off a cliff. More critically, the more information you pack in, the more the model loses focus; key instructions get drowned in the noise of conversation history. Context compression isn’t a question of whether to do it; it’s a question of how to do it well.
This article breaks down the complete engineering solution for Agent context compression. It’s not just summarization; it’s a long-task state management system.
The Core Tension
An Agent’s context window must hold multiple types of information simultaneously. System instructions define role and behavioral constraints. User goals describe the current task and deadline. Conversation history preserves multi-turn interaction context. Tool results contain API responses and file contents. Intermediate state records completed steps and pending items. External knowledge holds RAG-retrieved document fragments and reference material. When a task spans only 2-3 turns, fitting everything in is effortless. By turn 50 or 100, the total information volume far exceeds the window limit.
The information a long task requires exceeds what the model can hold. That’s the core tension. Compression is about maximizing the information density needed for the current decision within a limited window.
Layered Architecture
The intuitive approach is dumping conversation history, tool results, and external knowledge at the model all at once. The result: high cost, high latency, high noise. The correct approach is layered management, with different retention strategies for different information types.
| Layer | Name | Retention Strategy | Information Type |
|---|---|---|---|
| L1 | Hot zone | Verbatim, not a word changed | Last N conversation turns, current task state, incomplete steps |
| L2 | Warm zone | Structured summaries | Early conversation summaries, key conclusions from completed tasks |
| L3 | Cold zone | Indexes and references only | Historical tool results, old file contents, early search results |
| L4 | Frozen zone | Persisted storage, cross-session | User preferences, project conventions, historical lessons learned |
Hot zone information enters the model in full every time. Cold zone information carries only an index. Frozen zone information requires explicit recall.
Information transitions from hot to cold as the task progresses. Turn 1 stays in the hot zone verbatim. By turn 20, it moves to the warm zone as a summary. By turn 50, it moves to the cold zone as a reference only. User preferences go directly to the frozen zone for persistence. This flow is automatic and gradual, not a hard cutoff.
Compression Methods
Conversation Summarization
A common mistake when summarizing early conversations is producing a shortened version of the dialogue. A bad summary tells a story: what the user asked, what the assistant suggested, what was decided. A good summary preserves state: it records only completed decisions with their reasoning and a task list of what remains. An Agent with a good summary can resume work immediately; with a bad summary, it has to re-understand the entire context first. The principle: record conclusions, not process; record decisions, not discussions.
Tool Result Trimming
After an Agent calls a tool, the result can be enormous; a database query might return thousands of rows. Stuffing it all into context burns through the window in a turn or two. Trimming extracts field names and types, preserves key values and anomalies, keeps only Top-K plus aggregate statistics for list results, retains error codes and messages, and discards the rest while storing it in object storage with a reference ID.
For example, an Agent querying a user table gets 3,000 rows. After compression, it keeps the total count, field list, status distribution statistics, and a summary of the first 10 rows, plus a reference ID for later retrieval. From 5,000 lines down to under 20, with no critical information lost.
Deduplication
Agents frequently read the same file multiple times. Without deduplication, the same content appears several times in the context. The fix is simple: on the second read, keep only a reference pointing to the full content shown the first time. This step is remarkably effective in practice. According to Claude Code’s implementation (see this blog’s earlier article Context Management: The Secret of Four-Level Compression and Infinite Conversations), deduplication alone saves substantial tokens.
State Extraction
This is the most overlooked yet most critical compression technique. In multi-turn tasks, critical constraints scatter throughout the conversation: in turn 2, the user mentions the database is PostgreSQL; in turn 5, they confirm the API format is REST, not GraphQL; in turn 8, they mention the deployment environment is Kubernetes with memory limits. Once these get summarized away, they’re lost, and they’re hard constraints; losing them sends subsequent work in the wrong direction.
Before compressing, perform state extraction: organize scattered information into a structured state panel. The panel includes the current goal, a hard constraints list (database type, API style, deployment environment, deadline), completed steps, in-progress tasks, pending items, and current risks and blockers. The summary handles background; the state panel preserves critical constraints. Keeping them separate ensures compression doesn’t lose essential information.
Progressive Folding
Compression shouldn’t happen all at once; it should intensify gradually based on context usage. At 60% usage, no compression; everything stays verbatim. At 75%, run deduplication and tool result trimming; this is lossless compression. At 85%, start summarizing early conversations; this is lossy compression but preserves the state panel. At 95%, full compression: keep only the last 3 turns plus the state panel plus summaries.
Think of it like phone storage management: when space is plentiful, you keep every photo; when space gets tight, you delete videos first; when it’s nearly full, you delete unimportant screenshots. The more urgent the situation, the more aggressive the compression, but the state panel is always preserved.
Information Integrity Verification
How do you verify that critical information survived compression? That’s the real engineering challenge.
State Validation
After each compression cycle, run a state validation. Extract the critical constraints list before compression, execute the compression to generate summaries, then check whether all critical constraints appear in the summary. Specifically check: does the user goal still exist, are hard constraints intact, is the incomplete steps list complete, and is there anything that contradicts the original record? If validation passes, continue; otherwise, pull back the original text to correct the summary.
Conflict Detection
A more insidious problem than lost information is summaries contradicting original records. In turn 3, the user said MySQL; in turn 15, they changed their mind to PostgreSQL; the compressed summary only captured turn 3 and says “using MySQL”; the Agent continues working with the wrong summary, and every subsequent database operation goes in the wrong direction. Worse, errors accumulate: turn 20’s summary is based on turn 15’s summary, forming an error chain.
Four handling principles: original records take priority, when summaries conflict with original records, the original record wins; user confirmation takes priority, information the user explicitly confirmed gets the highest priority; mark conflict sources, when contradictions are found, mark which turn’s summary went wrong; trigger re-compression, when an erroneous summary is found, don’t patch it; recompress from scratch. Better to have the system slow down and recompress than let bad summaries propagate through the pipeline.
Replayability
Many systems only compress forward-looking, keeping the current state. In practice, looking backward matters equally. A task fails halfway through and you need to trace which step went wrong. A user says the previous approach was wrong and needs to revert. An Agent made a decision that turned out poorly and you need to review the decision process. All these scenarios require replay capability. If compression leaves only a summary, none of these scenarios work.
The correct approach is saving a lightweight checkpoint at every step. A checkpoint contains: current step number, user input, tool call records (with reference IDs pointing to original results), output summary, and how this step changed global state (what was completed, which files were modified). Original tool results aren’t in the checkpoint; it stores a reference ID, and you pull back the original data via that ID when replay is needed. During normal execution, checkpoints consume minimal tokens.
Integration with RAG
Context compression and RAG solve different problems. Context compression controls the volume of information entering the model; the core actions are trimming, summarizing, and layering; information sources are the conversation’s own history. RAG provides external knowledge the model doesn’t possess; the core actions are retrieval, reranking, and injection; information sources are external knowledge bases. The two must work together in a real system.
RAG retrieval results also need compression processing. If 20 fragments come back, each 2,000 tokens, that’s 40K tokens, potentially consuming the entire window. Compressed fragments must preserve metadata provenance: document name, version, update time, and permission level. What gets compressed is content, not provenance; every injected piece of information should trace back to its original source.
Evaluation Framework
The most common wrong metric is “how much did tokens decrease.” Compressing 90% but failing the task isn’t optimization; it’s information loss. The correct evaluation dimensions are: task success rate (can the Agent complete tasks correctly after compression), constraint retention rate (how many user-specified constraints survive compression), answer consistency (do pre- and post-compression answers match), latency change, and cost change.
Compression results on routine conversations are typically fine. What separates good systems from bad is edge cases. Long conversations (50+ turns) test whether the compression system maintains state coherence. Tool call failures test whether failure information is correctly preserved rather than compressed away. User corrections test whether corrected information survives. Constraint changes test whether compression reflects the latest requirements. Conflict scenarios test how the system handles contradictions between summaries and original records. Edge cases are the true test of a compression system.
The Complete Compression Pipeline
At the start of each conversation turn, the system executes the following flow: calculate current context usage and determine which compression stage applies; extract critical constraints from recent conversations and update the state panel; run compression as needed (deduplication, tool result trimming, conversation summarization, or full compression); compare pre- and post-compression constraint lists to check for omissions or contradictions; save this turn’s checkpoint (input, tool calls, output summary, state changes); finally, assemble the final context (state panel + last N turns verbatim + historical summaries + RAG injections) and send it to the model.
This pipeline runs at every conversation turn, but most steps are lightweight; heavy compression triggers only when context usage crosses thresholds.
What can be compressed is process, redundancy, and repetition. What cannot be compressed is decisions, constraints, and user intent. Draw that line clearly, and the compression system won’t have major problems.
Series navigation:
- Context Management: The Secret of Four-Level Compression and Infinite Conversations — Claude Code’s specific compression implementation
- Agent Memory System Design Specification — Frozen zone (external memory) design details
- Multi-Agent System Design Principles — Context management in multi-Agent collaboration