Every headline model of 2026 advertises a million-token context window: Kimi K3, DeepSeek V4, Inkling, Nemotron 3. Not long ago that number would have been a joke for local hardware, and our own KV cache explainer spelled out why: on a classic dense transformer, context is a memory tax that grows with every token. Run the math on a Llama 3.3 70B and it is brutal. Its cache costs about 0.66MB per token (80 layers, 8 KV heads, 128-dim heads, 16-bit), so 128k tokens of context is roughly 84GB, and a million tokens would be around 655GB, more memory than the model itself.
Yet here is a measured, real-world 2026 result: a 48GB Mac runs Qwen 3.6 27B at its full 262k context, model and cache together, with room left for macOS. Nothing about that machine changed. The models changed. While everyone was watching the MoE takeover, the same labs quietly made a second, matching bet: they rebuilt attention itself. This piece is about that second convergence, because it decides how much context your machine can really hold.
The problem: full attention pays rent on every token
In a standard transformer, every layer stores a key and value vector for every token you have ever fed it, and every new token looks back at all of them. That store is the KV cache. It grows linearly with context length, multiplied across every layer, and it lives in your fast memory alongside the weights. Double your context, double the tax. This design has perfect recall, and a price that made long context a datacenter luxury.
The 2026 releases attack that price from three directions, usually in combination:
1. Sliding windows: only remember what's nearby. Most layers watch only the last few thousand tokens, so their cache stops growing at the window size. Mistral 7B (Jiang et al., 2023) mainstreamed this in open models, using sliding window attention "to effectively handle sequences of arbitrary length with a reduced inference cost."
2. Linear attention: replace the cache with a fixed-size state. Instead of storing everything, a recurrent-style layer compresses history into a state whose size never grows, the lineage of Mamba (Gu & Dao, 2023), which showed "linear scaling in sequence length" with a constant-size state and roughly 5x inference throughput over comparable transformers. The modern refinement is Moonshot's Kimi Delta Attention (the Kimi Linear paper), a gated linear-attention module that cuts KV cache usage "by up to 75%" in long-sequence generation.
3. Keep a few full-attention layers as the safety net. Pure linear attention historically loses precise long-range recall, so the winning recipe is a hybrid: mostly cheap layers, with periodic full-attention layers preserving global lookback. Kimi Linear interleaves them 3:1; Inkling runs 55 sliding-window layers against 11 global ones.
Who is running what
| 2026 model | Attention design | What it does to context cost |
|---|---|---|
| Kimi K3 (2.8T) | Kimi Delta Attention (hybrid linear) | Cache/state stays in the tens of GB even at the 1M window |
| Qwen 3.6 27B | Hybrid: only 16 of 65 layers keep KV, rest are linear | ~4x less KV memory than a standard dense design, plus a fixed ~0.9GB recurrent state (owner-measured) |
| Inkling (975B) | 55 sliding-window + 11 global layers, banded position bias | Most layers' cache capped at the window, 1M supported |
| MiniMax M3 (428B) | MSA sparse attention | Vendor-reported 28.4x cut in per-token attention compute at 1M context, with up to 14.2x faster prefill on an H800 |
| DeepSeek V4 family | Compressed sparse attention (vendor description) | 1M context on both the 1.6T Pro and the 284B Flash |
| Kimi Linear 48B (the research vehicle) | KDA + full attention, 3:1 | Up to 75% KV reduction (paper-reported) |
Sources linked in each row and below; "owner-measured" and "vendor-reported" labels matter, and we keep them.
Set that table against the dense baseline from the intro and the shift is stark. The old question was "how much context can you afford?" The new question is "does your runtime understand the model's layout?", because on these architectures the cache stopped being the thing that runs out first.
What owners measure
The best public numbers come from the community member who converted Qwen 3.6 27B for llama.cpp, froggeric on r/LocalLLaMA, whose fit tables we referenced above: "only 16 of 65 layers use KV cache (verified). The other 48 are linear attention (fixed 898 MiB recurrent state). KV memory is ~4x less than a standard dense model."
His follow-on warning is the practical gotcha of this whole transition: "Runtimes that don't handle this (e.g. vllm) allocate KV for all 65 layers and show much higher memory usage." The architecture only saves you memory if your runtime knows about it. Support for each hybrid design lands in llama.cpp, vLLM, and MLX at different speeds, which is exactly why the Inkling and K3 GGUFs are still waiting on merged support (as we covered in the K3 hardware piece), and why a model can look "supported" while quietly allocating dense-sized caches.
The fine print: what 1M context still costs
Prefill did not get free. The memory tax fell, but reading a million tokens still takes real compute. As we covered in prompt processing vs generation, prefill is the compute-bound half, and on a prosumer box a truly full window is a wait measured in many minutes, sometimes worse. MiniMax's 14.2x prefill speedup claim exists precisely because this is the new bottleneck. Advertised context and patience-limited context are different numbers.
Recall is a spectrum now. A dense-attention model provably sees every token. A hybrid sees recent tokens sharply and older ones through a compressed state or periodic global layers. The Kimi Linear paper argues its hybrid matches or beats full attention on their evaluations, and vendors publish strong needle-in-a-haystack results, but independent long-context evaluation of the 2026 hybrids is still thin. If your workload depends on exact recall of something 800k tokens back, test it before you trust it.
Quantizing the cache still helps. On hybrid models the KV that remains can still be squeezed (q8_0 cache halves it again, the standard trick from our KV guide), and owner tables show exactly that combination reaching the full 262k on mid-tier boxes.
The decision cheat-sheet
| Your situation | What the attention rebuild means for you |
|---|---|
| Long documents / codebases on a 24 to 48GB box | Pick a hybrid-attention model (Qwen 3.6 class): six-figure context genuinely fits where a dense model managed ~32k |
| Agent sessions that grow all day | Hybrids keep memory flat as the transcript grows; watch prefill time, not VRAM |
| Exact recall over huge distances | Prefer models with more full-attention layers, and test your own needle case; hybrid recall is good but not guaranteed |
| Running brand-new hybrids | Check your runtime supports the architecture natively, or your "efficient" model allocates dense-sized cache |
| Dense favorites (Llama 3.3 70B class) | The old math still applies: budget ~0.66MB per token of context and quantize the cache |
Put the two 2026 convergences together and the design brief behind this model generation is complete: MoE froze the per-token compute cost while capacity exploded, and the attention rebuild froze the per-token memory cost while context exploded. Both bets favor the same hardware: as much memory capacity as you can buy, fed by decent bandwidth. Whatever box you own, the practical question is the same as ever, and our calculator now answers it with context length as a first-class input.
Sources and how we researched this
- Research spine: Gu & Dao 2023, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" (arXiv:2312.00752); Jiang et al. 2023, "Mistral 7B" (arXiv:2310.06825) for sliding window attention in open models; Moonshot AI 2025, "Kimi Linear: An Expressive, Efficient Attention Architecture" (arXiv:2510.26692) for Kimi Delta Attention and the 75% KV-reduction figure.
- Model architecture details: vendor model cards and announcements as covered in our linked per-model spotlights; Inkling's layer layout per the llama.cpp support PR; MiniMax MSA figures are MiniMax's reported numbers; DeepSeek V4's attention is the vendor's description.
- Owner measurements: froggeric's Qwen 3.6 conversion thread (verified KV-layer count, fixed recurrent state, per-tier context tables), quoted verbatim.
- The dense-baseline math is the standard KV formula (2 x layers x KV heads x head dim x bytes per value) applied to Llama 3.3 70B's published configuration.
- We have not benchmarked these architectures first-hand. This synthesizes the papers, vendor documentation, and attributed owner reports, cited inline.
Related: The KV cache, explained · Every frontier open model is a MoE now · Prompt processing vs generation · Bandwidth, Not TFLOPS · Speculative decoding, explained
