Midway through a long conversation with an AI, it forgets something you said at the beginning. You try to hand it a large file and get told it “exceeds the limit.” The mechanism behind these experiences—ones you encounter more often the more you use generative AI—is the context window.
The term appears in every new AI model announcement as a headline spec (“a 1-million-token context window”) and directly affects pricing and usability. This article draws on official Anthropic and Google documentation to explain what a context window is, why “bigger” is not automatically “better,” and how to work with the limits.
What Is a Context Window?
Anthropic’s official documentation defines the context window as “all the text a language model can reference when generating a response, including the response itself” 1.
The important point is that this is separate from the vast data the model was trained on. The same documentation explains that, unlike the training corpus, the context window “represents a ‘working memory’ for the model” 1. Google’s documentation likewise compares the context window to short-term memory and notes that there is a limit on how much information can be passed to a generative model 3.
In human terms, the training data is “the knowledge you’ve accumulated over your lifetime,” while the context window is “the amount of material you can spread out on your desk right now.” Material that doesn’t fit on the desk can’t be referenced, no matter how important it is. And as a conversation continues, past exchanges pile up on the desk: as Anthropic’s documentation describes, “as the conversation advances through turns, each user message and assistant response accumulates within the context window” 1. When a long conversation hits the limit, no new information can come in unless something is removed—which is exactly the “it forgot what I said” phenomenon from the opening.
Tokens: How Much Is One Million?
The size of a context window is measured in “tokens.” A token is the smallest unit an AI uses to process text—roughly, a chunk of a word or characters.
To get a feel for the scale, Google’s reference points are helpful. According to its documentation, one million tokens corresponds to roughly “50,000 lines of code (with the standard 80 characters per line),” “all the text messages you have sent in the last 5 years,” “8 average length English novels,” or “transcripts of over 200 average length podcast episodes” 3.
As of June 2026, the flagship models’ context windows have reached the million-token class. Google states that Gemini ships with a context window of one million or more tokens, noting that earlier generative models could only process 8,000 tokens at a time, with later versions reaching 128,000 3. Anthropic likewise states that Claude Fable 5 and Claude Mythos 5, as well as Claude Opus 4.8, 4.7, and 4.6 and Claude Sonnet 4.6, have a 1M-token context window on the Claude API (earlier models such as Claude Sonnet 4.5 have a 200k-token window) 1. These numbers change as model generations turn over, so check each vendor’s documentation for current values.
Bigger Isn’t Always Better: Context Rot
So should you just pick the model with the biggest context window and stuff in every document you have? It’s not that simple.
Anthropic’s documentation states plainly that “a larger context window allows the model to handle more complex and lengthy prompts, but more context isn’t automatically better,” because “as token count grows, accuracy and recall degrade, a phenomenon known as context rot” 1. The company’s engineering blog describes the same phenomenon: “as the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases” 2.
Why the degradation? The blog explains: “Like humans, who have limited working memory capacity, LLMs have an ‘attention budget’ that they draw on when parsing large volumes of context” 2. Even with a bigger desk, there’s a limit to how much material you can attend to at once. For this reason, Anthropic notes that “curating what’s in context [is] just as important as how much space is available” 1.
How to Work with the Limits
In practice, several techniques account for both the hard limit and the quality degradation.
The most prominent is RAG (Retrieval-Augmented Generation), which searches for and passes in only the relevant material. Instead of stuffing in every document, RAG puts just the needed portions into context—serving both to conserve the limit and to counter context rot. We explain how it works in our RAG explainer.
Another is summarizing and compressing conversation history. For long-running conversations, Anthropic recommends server-side compaction, which “automatically condenses earlier parts of a conversation, enabling long-running conversations beyond context limits” 1. In addition, some models such as Claude Sonnet 4.6 feature “context awareness”—the ability to track their remaining context capacity (token budget) throughout a conversation, enabling them to work on long tasks with their remaining space in mind 1.
This kind of management matters most for AI agents that run for long stretches while using tools, because every action piles tool results onto the context. How well context is managed can decide an agent’s performance. For the fundamentals of agents, see our AI agent explainer.
Sources
- Context windows - Anthropic official documentation
- Effective context engineering for AI agents - Anthropic official engineering blog (published September 29, 2025)
- Long context - Google official documentation (Gemini API)