Every Frontier Open Model Is a MoE Now. Here Is What That Does to Your Hardware Math

Line up every notable open-weight release of the past year and one pattern jumps out. Kimi K3: Mixture-of-Experts. DeepSeek V4: MoE. Inkling: MoE. GLM-5.2, MiniMax M3, Nemotron 3, gpt-oss, Qwen's whole current lineup, Gemma 4's larger sizes: MoE, all of them. The dense frontier model, the kind where every parameter works on every token, has quietly stopped being made at the top end. That is not trivia. It flips a rule that local-AI hardware buying was built on, and if you are choosing between a fast GPU and a big pile of memory in 2026, the sparsity takeover should decide the argument.

The lineup, in one table

A Mixture-of-Experts model splits its feed-forward layers into many small "experts" and routes each token through only a few of them, so two numbers describe it: total parameters (what you store) and active parameters (what computes per token). Our MoE explainer covers the mechanism. Here is what the current generation looks like on those two axes, using vendor-published figures:

Vendor-stated figures, linked in our per-model coverage and the sources below. NVIDIA's Nemotron-3-Ultra (550B) is MoE as well. Kimi K3's active count is not officially published; only its routing ratio is.

Read the columns separately and the design brief behind the whole generation appears. Totals span two orders of magnitude, 26B to 2.8 trillion. Active counts barely move: almost everything lands between 3B and 50B, and the frontier class clusters at 3 to 5 percent active. Model makers have effectively frozen the compute cost per token and are scaling the one axis that got cheap: stored parameters.

Why every lab converged on this

The economics were mapped out in the research years before the products arrived. The original 2017 MoE paper (Shazeer et al.) showed you could grow capacity over 1,000x with "only minor losses in computational efficiency." DeepMind's Unified Scaling Laws for Routed Language Models (Clark et al., 2022) made the key point precise: for routed models, "parameter count and computational requirement form two independent axes" of improvement. Knowledge scales with what you store; cost scales with what you compute. A lab that can add knowledge without adding per-token compute will do it every time, because inference is what they pay for at scale.

The recipe the 2026 models share traces to DeepSeekMoE (Dai et al., 2024): slice the experts fine (hundreds of small ones, pick several per token) and add a shared expert that always fires to hold common knowledge. Look back at the table with that in mind: Kimi's 384 or 896 experts, Inkling's 256 routed plus 2 shared, gpt-oss's 128 experts with top-4 routing. Same blueprint, different sizes. Mixtral proved the approach in the open in 2024, and the DeepSeek-V3 report demonstrated a 671B model trained for a fraction of dense-frontier cost. After that, the argument was over.

What it does to your hardware math

For a local runner, the two-axis design lands as one sentence: total parameters decide whether it fits, active parameters decide how fast it runs. Generation speed is memory-bandwidth-bound (our Bandwidth, Not TFLOPS piece covers why), and each token only reads the experts that fire, so a 428B model decodes like a ~23B one. But the router picks different experts every token, so all 428B must sit somewhere reachable. Capacity is the entry ticket; bandwidth only sets your speed once you are in.

Worked once with real numbers: GLM-5.2 stores 753B parameters, but each token reads roughly 40B of them. At a 4-bit quant that is about 22GB of weight-reads per token, so a 512GB Mac Studio moving 819 GB/s has a theoretical decode ceiling around 35 tokens per second on a model twenty times bigger than anything the box could hold dense. The stored size fell out of the speed equation; only the active slice and your bandwidth remain.

Owners see the same thing from the cheap seats. One B60 owner's published llama-bench runs put a dense 14B (Q8) at 24.7 tokens per second and a 23B-A3B MoE (Q4) at 63 on the same card, the same day. The quant difference explains part of that gap, but most of it is sparsity: the model with more total parameters decoded two and a half times faster because it reads far fewer of them per token.

That reweights the whole buying decision:

Capacity boxes win. The machines this generation favors hold a lot of memory at moderate bandwidth: a 512GB Mac Studio (runs 2-to-3-bit builds of the 753B-to-1T class in one box, as we covered with Inkling), 128GB unified-memory mini-PCs (gpt-oss-120b territory), or an ordinary desktop with a mid-range GPU and a big pile of system RAM.

The GPU + RAM split got a purpose-built tool. llama.cpp's --n-cpu-moe flag pushes the routed expert weights of N layers into system RAM while attention, KV cache, the router, and shared experts stay on the GPU, exactly the split MoE architecture invites, since the always-active parts stay in fast VRAM and the sparsely-touched experts sit in slow-but-huge RAM. The community offload guide is the reference. One caveat to keep straight: offload is a fit-enabler, not a speed-up. If the model fit in VRAM, offloading makes it slower. Its value is running a 117B or 284B total on a 12-to-24GB card at usable speed.

The floor dropped through the floor. The most-upvoted demonstration of the trend is a post from a builder in Burma running a 16B MoE on a 2018 dual-core i3 laptop with 16GB of RAM at ~9.6 tokens per second decode: "MoE is the 'Cheat Code': 16B parameters sounds huge, but it only calculates 2.4B per token." Dual-channel RAM and a sparse model, no GPU involved.

Mid-VRAM speed cards lost their pitch. The awkward tier is the 16-to-32GB GPU bought to run models that fit entirely in VRAM. The interesting open models no longer fit: the class that used to be a 30B dense download is now an 80B-to-284B-total MoE. A card in this tier is still excellent for the parts that stay hot (attention, KV, shared experts, prompt processing), but it now does its best work paired with 64 to 128GB of system RAM rather than standing alone.

The fine print

Three things keep this from being a free lunch. First, capacity got expensive at the worst moment: the memory price crisis means the RAM and high-capacity boxes this era rewards cost sharply more than a year ago. Second, active parameters are not a quality rating: a 35B-A3B is much stronger than a dense 3B (it stores 10x the knowledge) but it is not a dense 35B either; judge each model on evals and owner reports, not the ratio. Third, fast MoE models break some old speed tricks: as we found in our speculative decoding explainer, measured tests show speculation going break-even to slower on sparse models that are already quick, because every drafted token drags fresh expert weights into the verify pass.

The decision cheat-sheet

If one sentence should survive this article, it is the buying rule the trend leaves behind: in the MoE era, buy memory capacity first and bandwidth second, because the models will keep growing on exactly the axis that capacity serves. Every spec in the table above says the labs have already made that bet.

Sources and how we researched this

• Research spine: Shazeer et al. 2017, "Outrageously Large Neural Networks" (arXiv:1701.06538); Clark et al. 2022, "Unified Scaling Laws for Routed Language Models" (arXiv:2202.01169); Dai et al. 2024, DeepSeekMoE (arXiv:2401.06066); Jiang et al. 2024, Mixtral of Experts (arXiv:2401.04088); DeepSeek-V3 Technical Report (arXiv:2412.19437).
• Model figures: vendor model cards and announcements as covered in our linked per-model spotlights (Kimi K3, Inkling, GLM-5.2, MiniMax M3, Kimi K2.7), plus OpenAI's gpt-oss announcement (116.8B total, 5.1B active, 128 experts) and Google's Gemma 4 documentation.
• Offload mechanics: llama.cpp and the MoE CPU-offload guide.
• Community reports: the linked r/LocalLLaMA threads, quoted verbatim and attributed.
• We have not benchmarked these models first-hand. This piece synthesizes vendor documentation, the peer-reviewed literature, and attributed owner reports, cited inline.

Related: Mixture-of-Experts, explained · Bandwidth, Not TFLOPS · Unified memory, explained · Speculative decoding, explained · The hardware cheat sheet