The short answer: the NPU in your new laptop or mini-PC is a low-power chip built for small, steady AI tasks (background blur, voice, on-device features), and it is genuinely good at those. What it will not do is make your local LLM generate tokens faster. Token generation is limited by memory bandwidth, and an NPU does not raise that ceiling. Here is what an NPU is actually for, and why it is not the local-LLM accelerator the marketing implies.
This synthesizes vendor specs, the research literature, and owner reports, cited below.
What is an NPU?
NPU stands for Neural Processing Unit, a dedicated block on modern chips (Intel Core Ultra, AMD Ryzen AI, Apple, Qualcomm Snapdragon X) built to run neural-network math at very low power. Vendors advertise them in TOPS (trillions of operations per second), and the pitch is "AI acceleration built in." The key word is low power: an NPU is designed to handle a modest, continuous AI workload while sipping battery, not to blast through a large model as fast as possible.
What NPUs are genuinely good at
For the tasks they were designed for, NPUs are excellent, and they run them far more efficiently than a CPU or GPU would:
- Camera and audio features: background blur, noise suppression, eye-contact correction on video calls.
- On-device assistants and small always-on models that need to react instantly without spinning up the GPU.
- Image and photo features baked into the OS.
- Wake-word detection and other small models that run continuously in the background.
These share a profile: small models, steady load, battery-sensitive, latency-sensitive. That is the NPU's home turf, and it is a real benefit for laptop users even if it never touches a chatbot.
Why the NPU does not speed up your local LLM
Here is the part the spec sheets skip. Running a large language model has two phases, and the NPU helps neither in the way you would hope. As we cover in prompt processing vs generation, reading your prompt is compute-heavy, and generating each new token is memory-bandwidth-bound: to produce one token, the hardware must stream the model's active weights through memory. An NPU is tied to the same system memory as the rest of the chip, so it hits the same bandwidth wall. All its TOPS cannot help when the bottleneck is how fast weights can be moved, not how fast they can be multiplied.
The research backs this up. A 2025 study characterizing mobile chips for LLM inference, Chen et al.'s "Characterizing the Performance of Mobile SoCs for On-Device LLM Inference", found that single-stream token generation is bandwidth-bound across CPU, GPU, and NPU alike, and that the NPU cannot lift that ceiling. Owners report the same thing in practice: one builder who tried offloading prompt processing to an NPU found it was "actually worse than the iGPU." The NPU is not a slower path to the same place; for LLM generation it is often not the right unit at all.
So what runs your local LLM?
On the machines people actually use for local models, the work is done by the GPU (or the GPU portion of a unified-memory APU like Strix Halo or an Apple chip), fed by the widest, fastest memory available. The NPU sits mostly idle during LLM inference. This is why, when you shop a mini-PC or laptop for local AI, the NPU's TOPS figure is close to irrelevant, what matters is memory capacity and bandwidth, and GPU throughput. A big NPU number next to a narrow memory bus is a machine that will disappoint you at running models.
The bottom line
An NPU is a real, useful piece of silicon, for background AI features and efficient always-on tasks. It is not a local-LLM accelerator, and buying a machine for local models based on its TOPS rating is a mistake. Judge a local-AI machine on memory (how much, how fast) and GPU capability instead. When you do, run your model shortlist through our Can I run it? calculator, which sizes against the memory and bandwidth that actually decide the answer.
Sources and how we researched this
- Memory-bound decode and the NPU ceiling: Chen et al., "Characterizing the Performance of Mobile SoCs for On-Device LLM Inference" (arXiv:2501.14794, 2025), which finds single-stream generation bandwidth-bound across CPU/GPU/NPU.
- Why decode is bandwidth-bound: our Bandwidth, Not TFLOPS explainer and its cited roofline and inference-scaling literature.
- Owner reports on NPU prompt-processing offload from local-AI community threads. Vendor NPU specs and TOPS figures from Intel, AMD, Apple, and Qualcomm. This is a synthesis, not first-hand benchmarking.
Related: Prompt processing vs generation · Bandwidth, Not TFLOPS · Unified memory, explained · The M5 Mac's matmul hardware
