"1 million token context!" reads like a free upgrade. It isn't. A long context costs memory to store, bandwidth to re-read, and compute to ingest — and the bill grows with every token you add. Here's what you're really paying, and how to pay less. (This completes the trilogy with The KV Cache and Prompt Processing vs Token Generation.)
What "context window" means
The context window is the maximum number of tokens — your prompt plus everything generated so far — the model can attend to at once. 256K is roughly 200,000 words in play; 1M is a few books. The marketing usually stops there. The costs don't.
Cost 1 — Memory: the KV cache grows linearly
To avoid recomputing the past, the model caches a key and value for every token, in every layer — the KV cache. It grows with every token of context, on top of the model weights. For a typical 8B model in FP16:
| Context | KV cache |
|---|---|
| 8K | ~1 GB |
| 32K | ~4 GB |
| 128K | ~16 GB |
| 1M | ~128 GB |
At 1M tokens the cache alone dwarfs the model. (The full story.)
Cost 2 — Compute: prompt processing scales with length
Before the first token comes out, the model has to read your entire prompt — prompt processing (prefill), which is compute-bound. A 200K-token prompt takes far longer to ingest than a 2K one, so your time-to-first-token climbs with prompt size (PP vs TG).
Cost 3 — Bandwidth: generation slows as context grows
Every new token must stream the whole KV cache out of memory. The bigger the context, the more cache to read per token — so token generation gets slower the longer the conversation runs. Long context doesn't just risk running out of memory; it makes each token cost more to produce.
The "1M" asterisk: supported is not usable
A model accepting 1M tokens doesn't mean it uses them well. Recall degrades in the middle of very long inputs ("lost in the middle"), and — as the table shows — the memory to actually hold 1M tokens is impractical on a single GPU for most models. The advertised number is a ceiling, not a free lunch.
How to pay less
- MLA / GQA — attention variants that shrink the KV cache (Kimi K2.6 uses MLA, which is what makes its 256K context tractable).
- Quantized KV cache — store keys and values in 8-bit or 4-bit to halve or quarter the memory.
- Prompt caching — reuse the prefill of a repeated prefix (a system prompt, a long document) so you skip prompt processing on the next call.
- RAG instead of brute force — retrieve the few relevant chunks and put those in context, rather than stuffing in everything. Usually faster, cheaper, and more accurate than a giant window.
Bottom line
- A long context costs memory (KV cache), compute (prefill), and bandwidth (per token) — all rising with length.
- KV cache grows linearly: ~16 GB at 128K, ~128 GB at 1M for an 8B model.
- "1M context" is a ceiling, not free capability — recall and cost both bite.
- Reach for MLA/GQA, KV quantization, prompt caching, and RAG before reaching for a bigger window.