Plain-English definitions for the terms we use across Vetted Consumer, each with a deep-dive link where we've covered it properly. This page grows as the site does.
Active parameters · AWQ · Context window · Edge AI · GGUF · GPTQ · KV cache · llama.cpp · LM Studio · Memory bandwidth · Mixture-of-Experts (MoE) · NPU · Offloading · Ollama · Quantization · Tokens per second (tok/s) · Unified memory · VRAM
Active parameters
In a Mixture-of-Experts model, the subset of parameters that runs for each token, the rest sit idle in memory. Active parameters set a model's speed; total parameters set its memory footprint. A "35B-A3B" model needs 35B worth of memory but generates like a 3B model. Deep dive →
AWQ
Activation-aware Weight Quantization, a 4-bit format that protects the ~1% of weights that matter most to quality, identified by watching activations. Often matches or beats GPTQ accuracy at the same size, and is a first-class format in GPU serving engines like vLLM. Deep dive →
Context window
The maximum amount of text (in tokens) a model can consider at once, prompt, chat history, and reply combined. Bigger windows cost real memory, because the KV cache grows with every token of context. Deep dive →
Edge AI
Running AI models on small, low-power devices (single-board computers, embedded boxes) rather than desktops or servers, think an always-on 8B assistant at 25 W. Capacity is limited, but so are cost and power draw. Deep dive →
GGUF
The model file format used by llama.cpp and everything built on it (Ollama, LM Studio, Jan). Its superpower is flexibility: it runs on CPU, GPU, or a mix, and ships in many quantization levels (the Q4_K_M-style names). If you download a model file ending in .gguf, this is what you have. Deep dive →
GPTQ
A post-training 4-bit quantization method that uses second-order information to compress weights with minimal quality loss. GPU-only and all-in-VRAM, the long-time standard for GPU serving before AWQ arrived. Deep dive →
KV cache
The memory a model uses to remember the keys and values of every token in your context, so it doesn't recompute them for each new word. It grows linearly with context length and at long context can outweigh the model itself, the usual culprit when long prompts trigger out-of-memory errors. Deep dive →
llama.cpp
The open-source inference engine that made local LLMs practical on ordinary hardware. Runs GGUF models on CPU, GPU, or both; Ollama and LM Studio are friendlier layers on top of it. Deep dive →
LM Studio
A polished desktop app for downloading and chatting with local models, the easiest on-ramp to local AI. Built on llama.cpp (and MLX on Macs), with a GUI for model management and an OpenAI-compatible local server. Deep dive →
Memory bandwidth
How fast data moves between memory and the processor, in GB/s, the spec that determines token-generation speed, because the model's weights must be re-read for every token generated. For local LLM speed, bandwidth beats raw compute. Deep dive →
Mixture-of-Experts (MoE)
An architecture that splits a model into many "expert" sub-networks and activates only a few per token. You get big-model knowledge at small-model speed, but all the experts must still fit in memory. The reason 128 GB unified-memory boxes make sense. Deep dive →
NPU
Neural Processing Unit, a low-power accelerator built into modern laptop and phone chips for AI workloads. Today's NPUs mostly accelerate small on-device models and features; serious local LLM work still runs on the GPU or unified memory.
Offloading
Splitting a model between fast memory (VRAM) and slower memory (system RAM or even SSD), so models bigger than your GPU can still run, at reduced speed. llama.cpp's per-layer GPU offload is the classic example; MoE models offload especially well. Deep dive →
Ollama
A command-line-first local model runner built on llama.cpp: one command to download and serve a model, with a local API that other apps can use. The default choice for developers and self-hosters. Deep dive →
Quantization
Storing a model's weights at lower precision (e.g. 4-bit instead of 16-bit) to shrink memory and speed up generation, at a small quality cost. The single most important trick in local AI: it's why a 70B model fits in 40 GB instead of 140 GB. Q4_K_M is the community's default sweet spot. Deep dive →
Tokens per second (tok/s)
The standard speed measure for text generation. Watch for two different numbers: prompt processing (pp, how fast the model reads input) and token generation (tg, how fast it writes output). They stress different hardware, so always ask which one a benchmark means. Deep dive →
Unified memory
One pool of memory shared by CPU and GPU, as in Apple Silicon, AMD's Strix Halo, and NVIDIA's DGX Spark. It lets a machine hold far bigger models than any consumer GPU's VRAM, the defining feature of the current local-AI desktop class. Browse the category →
VRAM
A graphics card's onboard memory, the hard ceiling on what a discrete GPU can run. Model weights, KV cache, and overhead all have to fit. The first spec to check on any GPU for local AI. Deep dive →
Missing a term you had to look up elsewhere? Tell us and we'll add it. For how we research everything here, see our methodology.