Context Windows Aren't Free
When you call an LLM API, you're not just sending a message. You're purchasing context window space — and that space is finite, fast-filling, and billed per token.
128K context? 1M context? They all have a price per token, and that price compounds.
- Your prompt consumes tokens
- Your retrieved documents consume tokens
- The conversation history consumes tokens
- The LLM's output consumes tokens
The moment your context hits 80% capacity, two things happen: your bills spike unexpectedly, and the model's performance degrades — known as "lost in the middle" syndrome.
You are paying for every single token whether it contributes to the answer or not.
Truncation Has Consequences
When your context exceeds the limit, two things happen: (1) Latest messages get priority, (2) Everything in the middle gets cut. And what ends up in the middle? Often the most important context — medical records, legal clauses, the key document that answers the question.
You're not just losing tokens. You're losing the signal.
A 2024 Stanford study on RAG systems found that retrieval quality degrades significantly when context fills reach 70% capacity.
The surgical principle: Treat your context window like an operating theater — nothing extraneous, nothing wasteful.
Questions to ask before every API call:
- What is the minimum context this query actually needs?
- Where is my retrieval going wrong if the model can't find the answer?
- Am I confusing truncation with reasoning failure?
Today's Lesson
Context windows aren't buckets to fill — they're surgical instruments. Use only what you need, place it precisely, and measure the outcome. The developers who understand this outperform those who just throw more tokens at the problem.