AI Concepts & Fundamentals

Context Window

The amount of text an AI model can consider at once, measured in tokens.

Quick Answer

A context window is the maximum amount of text a language model can take in and consider at one time, measured in tokens (word-pieces). It covers both your input and the model's response. Anything beyond the window is cut off or must be summarized, so it limits how much of a document a model can reason over at once.

The Full Picture

The context window exists because a model doesn't have unlimited working memory. It processes a fixed span of text at once, and everything it 'knows' for a given answer has to fit inside that span — the prompt, any documents you paste, the conversation so far, and the reply it generates all share the same budget (IBM, 2024).

Size is measured in tokens, roughly three-quarters of a word each. Windows have grown quickly, from a few thousand tokens to hundreds of thousands in current models, which is enough for a long specification or a full contract. But it is still finite: exceed it and the earliest content is dropped or must be condensed, and the model simply can't see what fell outside.

Bigger isn't automatically better in practice. Larger windows cost more and run slower, and models can lose track of details buried in the middle of very long inputs — a well-documented 'lost in the middle' effect. Stuffing a huge document into the window is often worse than retrieving only the relevant passages, which is one reason RAG pairs so naturally with context limits.

For construction, the context window is why you can't simply drop an entire drawing set and specification manual into a chatbot and expect a reliable answer. The documents are often larger than the window, and accuracy degrades on the parts that do fit. Practical tools instead retrieve the relevant sections (via embeddings and a vector database) and feed only those into the window, keeping the model focused on what matters.

Real Examples

→Fits the window: A single 30-page spec section fits comfortably inside a large context window, so the model can answer questions across all of it at once.
→Exceeds the window: A full 800-page project manual is too large to load at once, so a tool must retrieve and feed only the relevant sections instead of the whole thing.
→Without AI vs with AI: Without managing context, pasting an oversized document silently drops the end; with a retrieval approach, only the passages relevant to the question are placed in the window.

Common Misconceptions

People assume: A bigger context window always gives better answers.

Actually: Larger windows cost more, run slower, and models often lose details buried in the middle of very long inputs. Feeding only the relevant passages usually beats dumping everything in, which is why retrieval is used alongside large windows, not replaced by them.

People assume: The context window is the model's long-term memory.

Actually: It's short-term working memory for a single request, not permanent storage. Once the request ends, the model doesn't remember what was in the window. Persisting knowledge across sessions requires external storage like a document store or vector database.

Does MeltPlan Solve This?

Not directly

A context window is a property of AI models, not a product. It matters in construction because project document sets routinely exceed what a model can read at once. MeltPlan handles this by retrieving and feeding only the relevant passages of your drawings and specs into the model rather than dumping whole sets in, with estimators verifying the output.

Frequently Asked Questions

What is a context window in an LLM?

The maximum span of text a language model can consider in a single request, measured in tokens. It includes your prompt, any documents provided, the prior conversation, and the model's response. Content beyond the limit is cut off or must be summarized.

What is a token?

A token is the unit a model reads text in — roughly a word-piece, about three-quarters of a word on average. Context window sizes and model pricing are both measured in tokens, so a 100,000-token window holds on the order of 75,000 words.

Does a bigger context window mean better results?

Not always. Larger windows cost more and run slower, and models can miss details buried in the middle of very long inputs. Retrieving and supplying only the relevant text often produces more accurate answers than filling a huge window with everything.

Why does the context window matter for construction documents?

Project drawing sets and specification manuals are often larger than any context window, and accuracy drops on lengthy inputs. So tools retrieve the relevant sections and feed only those to the model, rather than trying to load an entire document set at once.

Related Terms

Large Language Model (LLM)An AI trained on massive text that predicts and generates language.RAG (Retrieval-Augmented Generation)Grounding an LLM in real documents by retrieving relevant passages before it answers.Embedding (AI)Turning text or data into a vector of numbers that captures its meaning.Vector DatabaseA database that stores embeddings and finds the most similar ones fast.AI AgentAn AI that plans, uses tools, and takes multi-step actions toward a goal.Deep LearningMachine learning with many-layered neural networks that read raw images and text.

More AI Concepts & Fundamentals Terms

Sources

  1. IBM — What Is a Context Window?
  2. Zhao et al. — A Survey of Large Language Models (arXiv:2303.18223)
MELTPLAN