Three practical strategies for handling small context windows in LLMs are outlined: sliding window truncation, which caps conversation history as a FIFO queue for predictable token usage; token budgeting combined with RAG, which allocates fixed percentage zones for system instructions, chat history, and retrieved data; and briefer coverage of rolling summaries, prompt compression, and observation masking for specialized agent use cases. Python code examples illustrate the sliding window and token budgeting approaches, arguing that smaller, well-managed context windows can reduce latency and cost while avoiding the 'lost in the middle' problem.
Table of contents
IntroductionContext Truncation: Sliding WindowToken Budgeting and RAG (Retrieval-Augmented Generation)Beyond the Basics: Other StrategiesClosing RemarksQuestions this post answers
What is the sliding window technique for managing context windows in language models?
It treats conversation history as a FIFO queue, dropping the oldest exchanged messages as new ones arrive, so only a fixed number of recent turns (max_turns) are ever sent to the model. This keeps token usage and latency flat and predictable, at the cost of losing older context that falls outside the window. Developers tuning LLM memory strategies can find implementation walkthroughs like this on daily.dev.
How does token budgeting work when combining RAG with a small context window?
Token budgeting splits the available context window into fixed percentage zones with strict limits, for example allocating 20% to system instructions, 20% to chat history including the latest query, and 60% to retrieved documents. Insertion of retrieved data halts once its budget zone is filled, preventing large documents from crowding out other context and reducing the 'lost in the middle' problem. Teams designing RAG pipelines can track token-budgeting patterns like this via daily.dev.
What are alternative strategies to sliding windows for managing LLM context beyond the basics?
Rolling summaries use an auxiliary LLM to condense older conversation history into a compact paragraph, requiring extra API calls. Prompt compression strips filler words and redundant data via an algorithm without a second model call, but risks losing nuance if too aggressive. Observation masking hides older structural noise, like database queries or execution logs, useful in autonomous agent systems but harder to implement safely. Engineers weighing context-management trade-offs for AI agents can follow techniques like these on daily.dev.