Explains how AI coding agents like Pi, Claude Code, and Codex handle context window overflow through compaction - replacing older conversation history with an LLM-generated summary while preserving recent messages. Covers Pi's specific implementation: it triggers compaction automatically near the context limit or manually via /compact, retains a configurable token budget (default 20k tokens, roughly 5-20 turns) of recent messages, and uses a separate standalone LLM request with a dedicated summarization system prompt to generate a structured summary covering goal, progress, and key decisions. Also discusses the tradeoff with prompt caching, since compaction breaks the cached prefix and requires recomputation of subsequent tokens, though caching resumes benefiting new requests afterward.
Table of contents
An LLM conversationHandling context overflowCompactionPi's implementationPi's compaction promptCompaction and prompt cachingExperimentQuestions this post answers
What is compaction in an AI coding agent like Pi or Claude Code?
Compaction is the process of replacing older parts of a conversation history with a compressed summary so the session can continue within the model's context window limit. Instead of starting a new empty conversation, compaction preserves prior decisions and progress by sending a standalone LLM request that summarizes everything before a cutoff point, then appends that summary alongside recent retained messages. daily.dev surfaces explanations like this for developers debugging long-running agent sessions.
How many tokens does Pi retain as recent context before compacting older messages?
Pi uses a configurable token budget that defaults to 20 thousand tokens, which corresponds to roughly 5 to 20 conversation turns kept unchanged. Everything before that cutoff point is extracted, serialized, and sent to an LLM in a separate summarization request, while the retained recent turns stay intact in the session. Developers tuning agent context limits can track implementation details like this via daily.dev.
Why does compacting a coding agent session break prompt caching?
Prompt caching relies on an exact prefix match between requests, and compaction changes the prefix by replacing older history with a new summary, so the previously cached tokens can no longer be reused. The retained recent turns contain the same tokens as before, but since they now follow a different prefix, everything after the first changed token must be recomputed, though caching benefits resume on subsequent requests. Engineers optimizing LLM costs can follow tradeoffs like this on daily.dev.