The short answer: an embedding model is a second, much smaller model that turns text into a list of numbers (a vector) so a computer can measure meaning by distance. It is the piece that makes local RAG work: it finds the right chunks of your documents to feed your chat model. You run it alongside your LLM, it needs almost no hardware, and picking a good one matters more than most people expect. Here is what it does and which to use.
This is a plain-English synthesis of the retrieval literature and project documentation, cited below.
What is an embedding model, in plain English
Your chat LLM generates text. An embedding model does something narrower: it reads a piece of text and outputs a fixed list of numbers, often 384, 768, or 1024 of them, that represents the text's meaning. Two passages about the same topic land close together in this number-space; unrelated passages land far apart. That is the entire trick. Once every chunk of your documents is a vector, "find the passages most relevant to this question" becomes "find the nearest vectors," which a computer does instantly.
Why RAG needs one
Retrieval-augmented generation, the standard way to let a local model answer from your own files, has two models working together. The embedding model is the librarian: it indexes your documents and, at question time, retrieves the handful of most relevant chunks. The LLM is the writer: it reads those chunks and composes the answer. The approach was formalized in Lewis et al.'s RAG paper (2020), and the retrieval half rests on dense embeddings of the kind introduced in Dense Passage Retrieval (Karpukhin et al., 2020). If your local RAG setup gives vague or wrong answers, the culprit is often the embedding step feeding the LLM the wrong chunks, not the LLM itself.
Do embedding models need a GPU?
Barely. Embedding models are tiny compared to chat LLMs, typically a few hundred million parameters or less, so they run fast on a CPU and use a few hundred megabytes of memory. You can run one on the same machine as your LLM without noticing the load, or even on a mini-PC or a Raspberry Pi. The cost is not hardware; it is choosing a model whose vectors actually capture meaning well for your content.
Which local embedding model should you use?
A few families dominate local use, and the right choice trades size against quality:
| If you want... | Reach for |
|---|---|
| Tiny, fast, runs anywhere | A small all-MiniLM-class model (384-dim); great for a first RAG build |
| A strong general default | A BGE or E5-family model (base or large); the common local sweet spot |
| Best quality, more compute | A larger multilingual or instruction-tuned embedder (1024-dim) |
| Code search / retrieval | A code-specialized embedding model, not a general one |
Rather than trust a vendor's chart, check the MTEB leaderboard on Hugging Face, an open benchmark that ranks embedding models across dozens of retrieval and classification tasks. Filter by model size and language and pick near the top of what fits your machine. Two practical rules: match the embedding model's language and domain to your documents, and never mix embedders, everything in one vector index must be embedded by the same model, or the distances stop meaning anything.
The settings that quietly matter
Two knobs affect RAG quality as much as the model choice. Chunk size: split documents into passages of a few hundred tokens with a little overlap, too big and retrieval gets muddy, too small and it loses context. Number of chunks retrieved: pulling the top 3 to 6 chunks is usually right; more just crowds the LLM's context and can dilute the answer, which ties back to the context-length cost of stuffing too much in.
The bottom line
The embedding model is the unglamorous half of a local RAG stack, and the one people most often get wrong by ignoring it. It is cheap to run, quick to swap, and it decides whether your model sees the right information at all. Start with a solid BGE or E5 model, keep your chunks tidy, retrieve a handful at a time, and use the same embedder for your whole index. Then point your chat model at it and let the librarian do its job.
Sources and how we researched this
- RAG architecture: Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arXiv:2005.11401, 2020).
- Dense retrieval / embeddings: Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering" (arXiv:2004.04906, 2020).
- Model selection: the open MTEB embedding leaderboard and the sentence-transformers project documentation. This is a synthesis; we have not benchmarked every embedder first-hand.
Related: RAG on a local LLM, explained · The KV cache and context length · Which local runtime to use
