Local LLMs are fantastic, and they keep getting better at a staggering pace. I have non-negotiable reasons for preferring a local setup over relying on cloud giants like Claude or ChatGPT. Because of that, I’ve been relentlessly trying to integrate local models into my actual, day-to-day workflow.

But as anyone who has tried will tell you, the DIY route comes with distinct caveats and drawbacks.

The absolutely biggest hurdle is the context window. To get better responses from an LLM, we need to give it more context. But context is expensive. And as soon as the context window starts to fill up, the model starts to deteriorate in both speed and quality. So what are we supposed to do?

The problem, I realized, was not the LLM itself. The problem was that I was using the LLM for a job it was never meant to do. And the fix was a tiny model I had completely ignored.

LLMs and the problem of context

It really is expensive

Half of our experience with a language model comes down to what we put in front of it. If we give it a vague, thin prompt, you get a vague, thin answer. For better responses, we need to give the LLM more context.

The problem is that context is expensive. On a cloud model, more context means more tokens, and more tokens mean higher costs. Locally, it means slower inference, because every single token in that context window has to be processed during generation. For a local model specifically, this matters even more.

We are already working with smaller machines and tighter constraints. Dumping entire documents into the context window every time you ask a question actively degrades the experience. The model spends too much of its energy dragging around information, rather than actually reasoning with the parts that matter.

An embedding model is what let me fix that.

Embedding models

The unsung warriors of local LLMs

Credit: MUO

This 2D view is just for illustration. Real embedding use many more dimensions.

The tiny model in question is an embedding model. Embedding models come in different sizes, but compared to an LLM, they really are tiny. What an embedding model does is slightly abstract, but I’ll do my best to explain it.

An embedding model takes a piece of text and converts it into a long list of numbers. That list is called a vector, and it represents the meaning of the text in mathematical space.

Two sentences that mean roughly the same thing will produce vectors that are close together in that space. Two sentences that have nothing to do with each other will be far apart.

This is not keyword search. The model does not care about exact keywords in the same way a normal search box does. The embedding model understands semantic similarity.

If my explanation and chart still didn’t do the trick, the YouTube short below does the topic justice:

This pattern is called RAG

RAG or Retrieval-Augmented Generation. The idea is that instead of baking all your knowledge into the model itself, you keep it external, in a vector database, and retrieve only the relevant pieces at the moment you need them. The model’s job stays focused on what it’s good at, which is generating a response.

First, an embedding processes your documents and converts each chunk of text into a vector. Those vectors get stored in a dedicated database (something like ChromaDB or Qdrant, both of which run locally on your machine). This part happens once.

Then, when you ask a question, the same embedding model converts your query into a vector too. The database runs a similarity search and surfaces the chunks that are closest in meaning to what you asked. Those chunks get passed to your LLM as context, alongside the original question. The LLM handles the interpretation.

You have already benefited from this without knowing. NotebookLM, Google’s research assistant, lets you drop thousands and thousands of words worth of sources into it for free. Is Google being unusually generous with tokens?

No. That’s embedding at work. It doesn’t load all your sources into the model on every query — it retrieves only what’s relevant, cheaply and instantly. The heavy lifting is offloaded to a process that costs almost nothing to run.

My local setup with an embedding model

Yours can be very different, of course

Although the setup works beautifully, I don’t have beautiful screenshots to share as the output is just an LLM response.

The most immediate use case for a local LLM is your documents. For this, you can already demo the idea in LM Studio, the popular local LLM interface, without much setup. LM Studio already ships with RAG baked in. That is what the “embedding model updated” toasts are about. In a new conversation with your LLM, make sure the RAG MCP is enabled, then upload your documents. That is basically it.

I wanted to persistently embed my Obsidian journal entries, and this wasn’t persistent enough. For anything more permanent, I recommend downloading an embedding model and hosting it alongside your LLM.

From here, you have options. You can use a pre-built app like OpenNotebook, which is essentially a self-hosted NotebookLM clone, then point it at the LLM and embedding model you have hosted. From there, you upload your sources and use them inside that app.

But again, that requires you to use OpenNotebook for the queries. I wanted more flexibility.

So I set up Qdrant as the vector database. It is very easy to set up (thank you, Docker). Then, with a simple Python script, I can interact with the embedding model and have it embed my files into the Qdrant database.

From there, I packed all of this into an MCP server, connected it to OpenClaw, and that was it. Now my agent can pull in relevant context without burning through everything. The good thing about an MCP server is that I can plug it into almost every other AI interface. Claude Code, Codex, and Antigravity all support MCP servers.

Embedding goes way further than just “documents”

Use it for a persistent experience

Markdown is the programming language of 2026. Large language models take language as input and then output language. Words are the programming language.

If you have your own AI agent, then an embedding model is necessary for persistent memory. The alternative is sending every single past conversation as context in every new message. Obviously, that is not feasible at scale. It is slow, it breaks token limits, and it makes the experience worse. The embedding model is what allows the system to be selective.

So the workflow can be something like this: OpenClaw, or whatever agent you are using, keeps a record of conversations in Markdown. Those files are embedded into a vector database with an embedding model. Whenever OpenClaw needs to access past chats, its memory, it can query the vector database and get exactly what it needs to remember.

It is almost exactly how human memory works. You don’t start a conversation with your best friend by introducing yourself, telling them your name, and then listing everything you have ever done together (unless you’re making a point!) No. Those events are already there. They are recalled when needed.

I’ve used this for my journal, and it’s been a milestone progress in getting a local LLM to work with my Obsidian vault. Whatever your use case is, I suggest you set up the pipeline. Once the LLM stops carrying every piece of context on its back and starts retrieving only what matters, the whole experience shifts.