Context Window Strategy: Give AI the Right Information Instead of Everything

Longer context windows are one of the most visible signs of model progress. They make it possible to include large documents, conversation histories, codebases, and tool results in a single request. That capability can create a false sense of safety: if the model can see everything, surely it will find the right detail. In practice, more context can increase cost, latency, distraction, and contradiction. A long prompt is not the same as a well-designed information system. ## Define the decision the context must support Start with the output and evidence required. A model summarizing a contract needs clauses, definitions, exceptions, and effective dates. A support assistant needs the current policy and customer state, not every historic ticket. A coding agent needs the relevant module, interfaces, tests, and repository rules, not an unfiltered archive. Label information by role: instruction, authoritative source, reference, user-provided content, tool result, and background. Keep current and historical material distinguishable. When all text appears as one stream, the model may treat a quoted example or stale note as a live instruction. ## Retrieve before you summarize Retrieval should select evidence for the current task. Use metadata filters for tenant, region, product, status, and effective date before semantic ranking. Return source IDs, titles, sections, and compact excerpts. A second step can expand the most relevant sources when necessary. This often outperforms sending an entire folder into context. Test retrieval failure separately from reasoning failure. If the correct document is absent or ranked low, changing the prompt will not solve the problem. Track no-result cases, citation use, and source freshness. Keep a path for the user to inspect and correct the selected context. ## Use summaries with provenance Conversation summaries can save tokens, but they can also preserve an early misunderstanding. Store the summary version, source messages, date, and scope. Include unresolved questions and decisions separately from general narrative. When a current user instruction conflicts with a summary, prefer the current instruction according to a documented policy and ask for confirmation if the impact matters. For long documents, create hierarchical summaries that retain section references, key numbers, exceptions, and uncertainty. Do not compress a legal or safety document into a prose paragraph and assume all important conditions survived. Retrieve the original section for high-impact decisions. ## Order and budget the prompt deliberately Put the task, constraints, and output schema where the model can use them clearly. Separate untrusted content from policy. Avoid repeating the same instruction in slightly different words. Set a maximum input size and a per-workflow budget. If a task exceeds the budget, summarize, ask the user to narrow it, or hand it to a human; do not silently truncate the evidence. Measure attention problems. Test whether the model misses a key fact when it appears early, late, between irrelevant documents, or in a long table. Test conflicting sources and repeated claims. Use the results to design retrieval and presentation instead of assuming a larger window fixes everything. Context strategy is a product capability. The teams that organize information well will get more value from every model upgrade, while teams that dump everything into a prompt will pay more and trust less. Give the model enough evidence to decide, no more than the workflow can govern, and always retain a path back to the source. ## Add a context budget to the design Treat context as a budget with categories rather than a single token ceiling. Reserve space for the current request, authoritative policy, retrieved evidence, tool results, and the response. If tool output consumes the entire budget, the system should summarize or paginate it instead of silently removing the policy that governs the action. Record which items were omitted, compressed, or truncated so a reviewer can understand the answer's limits. Test the ordering strategy with adversarial examples. Put a critical exception at the beginning, middle, and end of a long context. Include two documents with different effective dates and one with a similar but irrelevant phrase. Measure whether the model selects the current source and cites it. These tests are especially important for agents that can act because a distracted answer can become an incorrect write operation. Give users a way to narrow scope. A date filter, repository branch, workspace selector, or document type choice often improves quality more than another prompt instruction. Show the selected sources and let the user remove one before asking the model to continue. Good context design is partly an interface problem: users should be able to see what the system knows before they trust what it says.