Context compression and compaction
For AI Agents: the Markdown version of this page is at https://ankole.agentbull.com/en-US/docs/context-compression-and-caching/index.md. The documentation index is at https://ankole.agentbull.com/en-US/llms.txt.
Any long-running conversation eventually reaches the model’s context window. To let threads continue cleanly past that limit, AIGateway automatically compacts the conversation history that each turn sees, implemented in ai_gateway/compaction*.ex.
Compaction is lossy by design, never silent: AIGateway replaces older turns with a structured summary, keeps recent turns verbatim, and persists a durable compaction artifact that the conversation points at. Once compacted, the older turns leave the model’s active prompt and the summary becomes the new reference state—normal continuation uses this checkpoint. Ankole retains predecessor rows so recovery can rebuild a local checkpoint when the provider binding changes or the user invokes /compress.
AIGateway history compaction
Ankole.AIGateway.Compaction owns automatic history compaction for stateful Responses conversations, governing when compaction fires, how the summary is generated, and what verbatim items survive alongside it.
The following sections describe local compaction, the default. Setting ai_gateway.compaction.upstream to true opts into provider-native compaction where supported, with local fallback; see AIGateway.
The trigger
AIGateway uses the latest provider-reported usage as its context-size anchor, then estimates the additional tokens from new input, tool results, tools, and instructions. It compares that projected total with the configured trigger and applies growth and compression-yield guards to avoid repeated low-yield compactions. Provider usage snapshots are not summed across turns.
Operators tune the trigger through the ai_gateway.compaction AppConfigure key:
| Setting | Default | Meaning |
|---|---|---|
threshold |
0.90 | the ratio of the model’s input context that triggers compaction |
max_threshold_tokens |
400,000 | an absolute cap on the computed trigger, so a very large context does not wait too long |
tail_rows |
4 | preferred number of recent history rows to retain alongside the summary |
user_message_budget_tokens |
20,000 | the token budget for replaying verbatim user originals |
When model context metadata is unavailable, Ankole uses 256,000 tokens. It subtracts any requested output budget, applies threshold, then caps the trigger at max_threshold_tokens. If the 64,000-token minimum would reach the effective context limit, it instead triggers at small_context_trigger_ratio (0.85) of that limit.
Token usage is not the only trigger. Encrypted reasoning and inline images take many bytes per token, so a history can grow large long before it crosses the token threshold. AIGateway also compacts a conversation when its stored history reaches three quarters of what a single tool-using response can carry (1,024 items or 8 MiB). This size trigger has no setting.
What the summarizer does
When local compaction is selected, AIGateway invokes the light model profile (falling back to primary) to condense the older span into a structured summary. The compaction prompt frames that output strictly as reference state, not instructions: "Do NOT continue the conversation. Do NOT respond to any questions. ONLY output the structured summary." The resulting summary captures user intent, key decisions, errors, and fixes while preserving exact file paths, function names, error strings, command lines, and identifiers—because paraphrasing a path or identifier breaks downstream tool calls.
AIGateway replays the summary alongside retained user originals and recent history as prior state rather than an open turn to answer.
What survives verbatim
Alongside the summary, AIGateway preserves two sets of items verbatim:
- Recent history rows (
tail_rows, default4) — a preferred tail size, adjusted to preserve tool call/output pairs and fit the context budget. - User originals —
CompactionRetentionreplays user text withinuser_message_budget_tokens; it may truncate text to fit. Compacted image payloads become bounded placeholders, while recent retained images remain intact.
These retained items help the model follow the task, while the summary remains a lossy account of older work.
The compaction artifact
Each compaction run writes a durable CompactionArtifact in AIGateway, anchoring the conversation’s visible history so subsequent turns continue forward from that checkpoint.
Compaction and cold-wake session rotation
ai_gateway.compaction and sessions.wake_recycle work together across warm and cold traffic:
- Warm active conversations (
ai_gateway.compaction): While turns arrive within the upstream provider’s prompt-cache window, Ankole keeps extending the same AIGateway conversation to maximize prefix-cache hit rates, compacting only when cumulative context crossesthreshold/max_threshold_tokensor the stored history reaches the size trigger. - Cold resumptions (
sessions.wake_recycle): When a Session resumes after sitting idle at leastmin_cold_gap_seconds(default3,600seconds) with context at or abovemin_context_tokens(default128,000tokens)—or afterim_max_idle_seconds(default259,200seconds / 72 hours) on an external IM channel—Ankole.Sessions.WakeRecyclecloses the cold AIGateway model conversation instead of replaying the full history, opens a successor branch, and injects a deterministic<wake-briefing>capped at8,000o200k_basetokens; see Sessions.
Tuning
- Raise
thresholdif your Agents mostly handle short threads and the default (0.90) compacts earlier than necessary. - Raise
tail_rowsif models lose track of immediate tool context right after compaction, trading summary headroom for more verbatim recent turns. - Raise
user_message_budget_tokensif long threads drop early user prompts from the compacted span and the model loses sight of original constraints.
All three fields live under ai_gateway.compaction in AppConfigure and apply starting with the next compaction rather than retroactively on the active turn.
What this guide is not
This page does not cover provider-side prompt caching—upstream KV/prefix caching belongs to each provider, configured via promptCacheKey on providers that support it. Nor does compaction preserve lossless prompt history, since older assistant turns leave the active context window by design. Finally, compaction is no substitute for clean task boundaries: while it lets a conversation run past a single context window, a thread that compacts every few turns usually belongs in separate Sessions or delegated Background Agent Jobs.
Next steps
- For the AIGateway concept page, read AIGateway.