Landmark 10 stops to get here · leads to 2

Context Window

The maximum number of tokens a language model can take into account at once, counting both the input it reads and the output it writes. Also called context length.

Your route here

10 stops · basics first
  1. Token ✓ understood

    The unit of text a language model reads and writes: a word, part of a word, or a character, drawn from the model's fixed vocabulary.

  2. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  3. Feature ✓ understood

    A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.

  4. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  5. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  6. Deep Learning ✓ understood

    A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.

  7. Representation Learning ✓ understood

    Letting a model learn its own internal description of the data, such as embeddings or layered visual features, instead of relying on features designed by hand.

  8. Embedding ✓ understood

    A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.

  9. Attention Mechanism ✓ understood

    A technique that lets a neural network weigh every part of its input when producing each output, focusing on the parts most relevant at that step.

  10. Transformer ✓ understood

    A neural network architecture, introduced in 2017, built from stacked self-attention and feed-forward layers; the basis of nearly every modern large language model.

  11. Context Window · you are here ✓ understood

Picture it

CONTEXT WINDOW · ILLUSTRATIVEsystem promptconversationdocumentsoutputlimitoverflow
Notice how system prompt, conversation, documents and output all share one fixed budget, and anything past it overflows.

A context window is everything a language model can see while producing its next token. Your system prompt, the conversation so far, pasted documents, tool results and the model’s own reply all have to fit inside one fixed budget, measured in tokens. Anthropic’s documentation describes it as the model’s “working memory”, as opposed to the much larger body of text it was trained on.

Nothing outside the window exists for the model. It has no memory between requests. A chat app feels continuous because it resends the whole history every turn, and when that history gets too long, something has to be dropped or summarized.

How big

Sizes have grown fast. GPT-3, in 2020, had a context window of 2,048 tokens, enough for a few pages of text. As of 2026, Anthropic lists current Claude models at up to 1 million tokens, room for a sizeable codebase. The diagram shows why a big number still runs out: the input and the output draw from the same budget, so a prompt that fills the window leaves no room for an answer.

Why there’s a limit

A transformer uses attention to let every token look at every other token. For a sequence of n tokens, that’s on the order of n² comparisons per layer, so doubling the context roughly quadruples the attention work. Serving long contexts also means holding cached state for every token in memory. Techniques like FlashAttention make long windows practical, and positional encodings have to be designed to cope with lengths the model rarely saw in training.

The catch

A bigger window doesn’t mean the model uses all of it well. In a 2023 study, models answered questions best when the relevant passage sat near the start or end of the input, and noticeably worse when it was buried in the middle. Anthropic’s docs call the general decline with length “context rot”. Longer inputs also cost more, since you pay for every token on every call.

So the skill is choosing what goes in, not just fitting more. Retrieval-augmented generation fetches only the passages that matter. Agent systems hand subtasks to helpers with fresh, smaller contexts. The Duel lets you watch a single model’s window overflow and see what splitting the work saves.

Where it sits

Context Window

Explore nearby

In the research

All papers →

12 papers that build on Context Window ; showing 5, canon first.

Frontier · Nov 2025 · 1.8K citations Qwen3-VL Technical Report It is the most-cited paper of the year and a reference point for open multimodal models. Frontier · Oct 2025 · 236 citations Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models It gives a principled shape to “context engineering”, the craft behind most agent harnesses. Frontier · Dec 2025 · 232 citations Memory in the Age of AI Agents A shared vocabulary for agent memory, one of the least settled parts of agent design. Frontier · Oct 2025 · 159 citations DeepSeek-OCR: Contexts Optical Compression It raises a new idea for long context: compress old history visually instead of keeping every text token. Frontier · Oct 2025 · 116 citations Kimi Linear: An Expressive, Efficient Attention Architecture It is a credible claim that linear attention can replace full attention without a quality penalty.

Sources

  1. Brown et al., "Language Models are Few-Shot Learners" . 2020 (GPT-3); section 2.1, context window of 2048 tokens
  2. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" . 2023
  3. Anthropic, "Context windows" . Claude API documentation, accessed September 2026