Context Rot

Context rot is the degradation in an LLM's retrieval accuracy and instruction-following as its context window fills, even when the window is nominally large enough to hold everything. The term comes from Chroma's July 2025 technical report "Context Rot: How Increasing Input Tokens Impacts LLM Performance", which evaluated 18 models including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, and found that models do not process context uniformly: the ten-thousandth token is not handled as reliably as the hundredth, and performance falls off as input grows even on tasks that stay trivially simple.

Two effects drive most of it. The first is position: Liu et al.'s 2023 "Lost in the Middle" study found accuracy highest when the relevant passage sits at the beginning or end of the input and lowest when it sits in the middle, a U-shaped curve that persists in models explicitly built for long context. The second is distractor sensitivity. Chroma's report found that a single semantically similar but incorrect passage measurably reduces accuracy relative to a clean input, that four distractors compound the effect, and that the damage is non-uniform, with particular distractors dominating the hallucinated answers across models. The report also found the degradation worsens as the question and the answer become less lexically similar, and noted that model families fail differently, with Claude models more likely to abstain under ambiguity and GPT models more likely to answer wrongly. The reason this surprised people is that Needle in a Haystack, the benchmark most vendors cite for long-context capability, is a literal string-match task that frontier models saturate, so near-perfect NIAH scores say almost nothing about retrieval under realistic conditions.

The conclusion for anyone building a RAG pipeline or an agent is direct: a bigger context window is not a substitute for retrieving less and retrieving better. Pasting fifty retrieved chunks into a million-token window because the window can hold them makes the answer worse than passing the five chunks that actually matter, because the other forty-five are distractors by construction, and reranking exists to remove exactly them. The work that pays off is unglamorous: retrieve broadly, rerank, cut hard, and put the highest-confidence evidence at the start or end of the prompt rather than in the middle. Chunking strategy matters for the same reason, since a chunk that splits the answer across a boundary forces the model to reassemble it from two positions.

This also changes how to read a vendor's long-context claim. A stated context limit is a capacity number, not an accuracy number, and the same applies to extraction providers: an API that returns a clean 3,000-token Markdown page is more useful than one that returns 40,000 tokens of navigation and boilerplate around the same article, which is the practical argument for content extraction over raw HTML. Firecrawl, Jina AI Reader, and search APIs such as Exa that return page content rather than link lists are all, in this framing, doing token reduction as much as data collection.