Pillar guide
Context rot
A working definition of context rot, what the Chroma research measured across 18 models, how it relates to the lost-in-the-middle position effect and to context loss in production agents, and the harness engineering that prevents it.
Definition
What is context rot?
Context rot is the tendency of a large language model's performance to degrade as its input context grows longer. Models do not process the 10,000th token as reliably as the 100th: recall, reasoning, and instruction-following decline unevenly as tokens accumulate, even on simple tasks. The term was popularized by a 2025 Chroma technical report that measured the effect across 18 leading models.
The phrase comes from a July 2025 technical report by the Chroma research team — Context Rot: How Increasing Input Tokens Impacts LLM Performance — that put a name on something practitioners had been reporting informally for a while: models that ace million-token retrieval benchmarks still get measurably worse at simple tasks as the input grows. Anthropic's engineering guidance adopted the term soon after, defining it as the observation that as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases.
This guide covers what the research actually measured, how context rot relates to the older lost-in-the-middle position effect, how it differs from context loss — the behavioral failure you observe in a production conversation — and what prevention looks like for teams running an agent in production. The short version of that last part: you cannot patch the model, and a bigger context window does not fix it. Every mitigation that works is a change to the agent harness — what goes into the window, what gets compacted, what moves to memory or retrieval, which tool outputs are worth their tokens. That makes context rot a harness engineering problem, and this page treats it as one.
Updated
The research
What the Chroma context rot research measured
The Chroma report is the source to read first, and its design is why the term stuck. Instead of letting task difficulty grow along with input length — the flaw in most long-context benchmarks — it holds each task fixed and deliberately simple, varies only the number of input tokens, and measures what happens across 18 models, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3.
- Performance degrades as input length grows, even on tasks the same models handle near-perfectly at short lengths — and the degradation is non-uniform rather than a smooth decay.
- The lower the semantic similarity between the question and the answer buried in the context, the earlier performance falls off as length increases.
- Distractors — text topically related to the answer but not quite it — hurt more as the context gets longer, and model families fail differently: GPT models showed the highest rates of confident hallucination, while Claude models tended to abstain under ambiguity.
- The surrounding text itself matters: across all 18 models, performance was consistently better on shuffled haystacks than on logically coherent ones — evidence that models are sensitive to the structure of the context, not just its length.
- On LongMemEval, a conversational question-answering benchmark, every model scored significantly higher on a ~300-token focused input than on the full ~113k-token chat history containing the same relevant information.
The report's conclusion is not that long context is useless. It is that whether relevant information is present in the context is not all that matters — how that information is presented matters more, which the authors name as the case for context engineering. Real workloads are far messier than the report's deliberately minimal tasks, so the measured effect is a floor, not a ceiling.
In production
LLM context rot in production agents
Benchmarks understate the problem for agents, because agents are the worst case for input length: every turn, tool call, and tool result accumulates in the window, and a session that starts sharp is operating on a very different context by turn forty.
The mechanics compound quietly. Tool outputs are the usual bulk — one verbose API response or file dump can be tens of thousands of tokens, most of which is never referenced again. Multi-step workflows stack instructions from earlier phases that no longer apply. Retrieved documents pile on top of conversation history. Nothing errors: every call in the trace returns 200 and latency looks normal. The degradation shows up only in behavior — the agent re-asks for information the user already gave, contradicts an earlier turn, or follows the most recent instruction while quietly dropping the constraint from turn three.
Anthropic's guidance on context engineering explains why this is architectural rather than incidental: every token attends to every other token, so the attention budget is stretched across a number of pairwise relationships that grows quadratically with input length. Context is a finite resource with diminishing marginal returns, and agents — which Anthropic describes as generating the very data that fills their own windows — deplete it turn after turn by construction.
SourceAnthropic, Effective context engineering for AI agents
Position effects
Lost in the middle: position matters as much as length
Two years before the Chroma report, Liu et al.'s Lost in the Middle: How Language Models Use Long Contexts documented the position half of the problem: models are best at using information at the very beginning or very end of the input, and measurably worse when the same information sits in the middle.
The paper tested multi-document question answering and key-value retrieval and found a U-shaped performance curve — a primacy bias toward the start of the context and a recency bias toward the end — with performance degrading significantly when models must access relevant information in the middle of long inputs. The effect held even for models explicitly built for extended context.
Context rot and lost-in-the-middle are two halves of the same engineering reality: how much you put in the window matters, and where it sits matters. Both break the naive mental model of a context window as random-access memory that merely has a capacity. The practical consequences follow directly — burying load-bearing instructions in the middle of a long system prompt is a design bug, and appending forever instead of compacting guarantees that important early turns migrate into the low-attention middle.
SourceLiu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172)
Disambiguation
Context rot vs. context loss
The two terms get used interchangeably and should not be. Context rot is a model-level mechanism: degraded ability to use what is in the window as the window grows. Context loss is the conversation-level failure you actually observe: the agent forgets earlier turns, contradicts itself, or asks the user to re-supply information it was already given.
The distinction matters because context loss has more than one cause, and context rot is only one of them. An over-aggressive compaction step that summarizes away a key constraint, a memory store that fails to retrieve the relevant fact, a truncation bug in the harness, or a session boundary that silently drops state will all produce identical symptoms with the model working perfectly. Treat every context loss as context rot and you will reach for a bigger window when the actual fix is a better compaction prompt — and the bigger window will make the rot worse.
That is why detection and attribution have to be separate steps. Moda detects context loss as a first-class behavioral failure across production conversations — one of six categories in its behavioral failure taxonomy, alongside tool misuse, reasoning loops, goal drift, hallucinations, and agent laziness — and attributes each detection to the harness component responsible before anything gets changed.
Prevention
How to prevent context rot
You cannot prevent context rot inside the model, and you cannot buy your way out with a bigger window — the research shows degradation well before advertised limits. Prevention is context engineering, and every technique that works is an edit to a specific harness component.
- Curate what enters the window. Retrieval that pulls the relevant slice just-in-time beats preloading entire corpora; the LongMemEval result — a focused input beating the full history containing the same information — is the direct measurement of this.
- Compact instead of appending. Summarize completed workflow phases and older turns, keeping decisions and constraints while dropping raw transcript — and treat the compaction prompt itself as a harness artifact that can silently destroy information when badly tuned.
- Trim tool outputs. Verbose tool results are the fastest-growing source of dead tokens in agent sessions; return the fields the agent needs, not the whole payload. Tool schemas and response shaping are harness components.
- Move durable state to memory. Facts that must survive across turns and sessions belong in an explicit memory store, written and retrieved deliberately, not in an ever-growing transcript.
- Isolate deep sub-tasks in sub-agents. A focused sub-agent with a clean window returns a condensed result to the orchestrator, so heavy exploration never permanently pollutes the main context.
- Position deliberately. Load-bearing instructions go at the start or end of the window, never buried mid-prompt — the lost-in-the-middle result applied as a design rule.
Every item on that list lives in the harness — prompts, tools, skills, evals, memory — not in the model. That is the practical answer to how to prevent context rot: engineer the harness so the model never has to fight a rotten context, and verify each change against evals built from your real traffic rather than trusting that a new compaction prompt or retrieval policy did no harm.
The fix
Context rot is a harness failure
A failure you cannot fix in the model, whose symptoms appear in production behavior, and whose every remedy is a prompt, tool, skill, eval, or memory change belongs to the harness. Treating context rot as a model limitation to wait out — or as a metric to watch — leaves the one layer you control untouched.
Watching a context-length metric tells you the window grew; it does not tell you which conversations degraded, which harness component is responsible, or whether your fix worked. The loop that does: detect context loss across the production population, attribute each detection to the component responsible — this compaction prompt, this verbose tool schema, this memory store that failed to retrieve — ship the change, and verify it against replayed traffic before it counts as fixed.
That loop is what Moda runs as a product, built for teams with an agent already in production: production traces in over OpenTelemetry, context loss and the other behavioral failures detected across every conversation, each finding attributed to the specific harness component responsible, and each candidate improvement verified against replays of real traffic before it ships. The demo is the fastest way to see it against your own traffic.
Frequently asked
Questions
What is context rot in AI?
Context rot is the degradation of a large language model's performance as its input context grows longer. Even on simple tasks, recall and reasoning decline as tokens accumulate — non-uniformly, and differently across model families. The term was popularized by Chroma's 2025 technical report, which measured the effect across 18 models, and adopted by Anthropic's engineering guidance, which defines it as the model's decreasing ability to accurately recall information from the context as the context grows.
What causes context rot in LLMs?
The architecture. Transformer attention relates every token to every other token, so the attention budget is stretched across a number of pairwise relationships that grows quadratically with input length, and models see far less long-sequence data in training than short-sequence data. The measurable consequences: lower needle-question similarity fails earlier as length grows, distractors hurt more at length, and even the structure of the surrounding text changes outcomes.
Is context rot the same as lost in the middle?
They are related but distinct findings. Lost in the middle (Liu et al., 2023) is a position effect: models use information at the beginning or end of the context better than the same information in the middle, producing a U-shaped performance curve. Context rot (Chroma, 2025) is a length effect: performance degrades as total input grows, wherever the information sits. In practice both apply at once — a long context degrades overall, and its middle degrades most.
Is context rot the same as context loss?
No. Context loss is the behavioral failure you observe in a conversation — the agent forgets earlier turns, contradicts itself, or re-asks for known information. Context rot is one mechanism that produces it. Context loss can equally be caused by over-aggressive compaction, failed memory retrieval, or truncation bugs in the harness, with the model performing perfectly. Diagnosing which one you have determines the fix, which is why detection and attribution are separate steps.
How do you prevent context rot?
By engineering the context, not by buying a bigger window. The techniques that work: retrieve the relevant slice just-in-time instead of preloading everything, compact completed turns and phases into summaries, trim verbose tool outputs, move durable facts into an explicit memory store, isolate deep sub-tasks in sub-agents with clean windows, and place load-bearing instructions at the start or end of the window. Every one of those is a harness change — a prompt, tool, skill, or memory edit — and each should be verified against evals built from real traffic.
Does a bigger context window fix context rot?
No. The Chroma results show degradation begins well before advertised limits and is non-uniform — the marginal token is not free just because the window officially has room for it. A bigger window raises the ceiling on how much rot you can accumulate; it does not change the mechanism. Teams that handle long-running sessions well spend their effort on what enters the window, not on how big it is.
How does Moda help with context rot?
Moda is a harness engineering platform for teams with an agent already in production. It detects context loss — the production symptom of context rot — across every conversation as part of a six-category behavioral failure taxonomy, attributes each detection to the harness component responsible (a compaction prompt, a verbose tool schema, a memory store that failed to retrieve), and verifies each candidate fix against replayed production traffic before it ships. Production traces in, verified harness improvements out.
For AI agentsThis page as Markdown · llms.txt · Agent skills index · Claude Code skill
Find where context rot is costing your agent.
Moda is a harness engineering platform: it detects context loss across production conversations, attributes each failure to the prompt, tool, skill, eval, or memory component responsible, and verifies the candidate fix against replayed traffic before it ships.