The short answer: a slow local LLM almost always comes down to one of six things, and they have a rough order of impact. In descending order: the model does not fit in fast memory (so it spills to disk or slow RAM), your memory bandwidth is low, the model is too big or too dense for your hardware, prompt processing is dragging on long inputs, you are on the wrong runtime or backend, or your settings are off. Work down this list and you will find it. Here is how to diagnose each.
This synthesizes the performance literature and owner reports, cited below.
1. The model does not fit in fast memory
This is the number-one cause of "why is it crawling," and it is a cliff, not a slope. If the model plus its KV cache does not fit in VRAM (or unified memory), the runtime spills the overflow to system RAM or, worse, disk, and speed collapses from dozens of tokens per second to a fraction of one. A reviewer watched a model that did not fit drop to 1.6 tok/s where a fitting model ran at 11+. Fix: use a smaller quant so the whole model fits, or offload a Mixture-of-Experts model's experts to RAM deliberately (fast) rather than letting it spill to disk (catastrophic).
2. Your memory bandwidth is low
Once the model fits, generation speed is set almost entirely by memory bandwidth: each token requires streaming the active weights through memory. This is why a single stick of RAM (half the bandwidth of dual-channel) chokes generation, why slow DDR4-2400 underperforms DDR5, and why a 24GB GPU at 936 GB/s decodes far faster than a 128GB mini-PC at 256 GB/s. Fix: fill all memory channels, buy the faster kit, and when comparing machines, compare bandwidth, not TOPS or core counts.
3. The model is bigger or denser than your hardware wants
Generation speed scales with the active parameters read per token. A dense 30B model reads all 30B; a sparse 30B-A3B Mixture-of-Experts reads only 3B, so it decodes roughly ten times faster on the same hardware. If a model feels sluggish, a modern MoE of similar or larger total size may actually run faster. Fix: prefer sparse MoE models for your tier, and do not assume a smaller dense model is quicker than a bigger sparse one.
4. Prompt processing is the real delay (long inputs)
If the model is fast once it starts but takes forever to begin responding, your bottleneck is prefill, not generation. As we cover in prompt processing vs generation, reading a long prompt (a big document, a full codebase, a long agent history) is compute-heavy, and it is exactly where unified-memory machines historically struggled, some owners reported 10-plus-minute waits before the first token on large contexts. Fix: shorten your context, use a machine with stronger prefill compute (the new M5 Macs added matmul hardware for exactly this), or add a discrete GPU for the prefill step.
5. You are on the wrong runtime or backend
The same model, same hardware, can run 2 to 5 times faster on a different runtime. As our runtime guide lays out, Ollama is convenient but often slower than a tuned llama.cpp; on AMD hardware, the Vulkan backend frequently beats ROCm; on Apple silicon, MLX can beat a generic build. And most people never turn on speculative decoding, a free 1.5-to-2x on the right workload. Fix: try llama.cpp directly, test the alternate backend for your GPU, and enable speculative decoding for deterministic tasks.
6. Your settings are working against you
Smaller effects, but real. A KV cache set to full 16-bit at a huge context wastes memory you could spend on the model, quantize it to 8-bit. Some runtimes default to a low GPU-layer count, leaving work on the CPU. And a runaway context length silently eats memory and slows everything. Fix: push as many layers to the GPU as fit, quantize the KV cache, and keep context to what you actually need.
The diagnostic order
| Symptom | Most likely cause |
|---|---|
| Painfully slow, near unusable (<2 tok/s) | Model does not fit; it is spilling to disk (#1) |
| Steady but underwhelming tok/s | Low memory bandwidth (#2), or dense model (#3) |
| Fast once started, but long wait first | Prompt processing on long input (#4) |
| Slower than others report on same hardware | Wrong runtime/backend (#5) or settings (#6) |
The one habit that solves most of this: measure your own tokens per second and note where the delay is (before the first token, or during generation), because that single observation points straight at the cause. Start by confirming your model fits in fast memory, our Can I run it? calculator tells you the largest quant that fits and the tok/s to expect, so you know whether you have a fit problem or a tuning problem.
Sources and how we researched this
- Memory-bound decode: our Bandwidth, Not TFLOPS piece and its cited roofline and efficient-inference literature.
- Prefill vs decode: our prefill explainer, drawing on the SARATHI and Splitwise serving papers.
- Runtime and MoE effects: our runtime guide and MoE explainer, plus attributed owner reports. This is a synthesis, not first-hand benchmarking.
Related: Bandwidth, Not TFLOPS · Speculative decoding, explained · The quantization guide · Every frontier open model is a MoE now
