Article · Blog
The embedding you generated doesn't represent what the model will actually read
Most RAG systems fail before the search even happens. The problem is in how content was turned into vectors, not in how it was retrieved. Indexing the wrong text is a silent error that only shows up in the final response.
The embedding you generated doesn't represent what the model will actually read
I spent a few months building a context retrieval system for a product that used technical documentation as its knowledge base. The search worked. The chunks came back. The model responded.
Except the responses were wrong often enough to be a real problem, not an edge case.
I'd look at the retrieved chunks and they seemed relevant. I'd look at the query and it was clear. The model was capable. But the final response kept living somewhere between what was asked and what was in the documentation, without fully reaching either.
It took me longer than it should have to find where the error was.
The problem wasn't in the search
The first instinct when debugging a RAG system is to tweak the search. Change the similarity function. Increase top-k. Try MMR instead of pure similarity. Re-rank the results.
I did all of that. It improved marginally. The problem persisted.
The error was upstream. It was happening at the moment the content was turned into a vector.
Embedding captures semantics. But semantics of what, exactly? Of the text you passed to the embedding function. And that text is rarely the same as what the model will actually need to read to answer the question.
That sounds obvious written out like this. It's not obvious when you're in the middle of a pipeline.
What most people index vs. what the model needs
In a technical documentation system, the most common mistake I made, and saw repeated in other projects, was indexing continuous blocks of text extracted directly from the document.
The problem is that technical documentation was written to be read with context. A paragraph explaining some behavior assumes you've already read the section heading, the subheading, the previous paragraph. When you take that paragraph in isolation and generate an embedding, the vector represents a fragment that, stripped of its original context, means something different.
The search will retrieve that chunk because it's semantically close to the query. But the model will receive text that starts mid-thought, with no sense of where it sits in the document's structure, without the heading that qualifies what's being explained.
The model's response will be good enough to look correct, and wrong enough to cause problems in production.
The asymmetry you're ignoring
There's a fundamental asymmetry in RAG systems that most people don't address explicitly: the embedding is generated once, at indexing time. The query is generated at search time. And the text the model will read is the original chunk, not its vector representation.
These three things need to be aligned. They rarely are.
The query arrives in natural language, usually as a question. The chunk's embedding was generated from affirmative text, possibly technical, possibly fragmented. Vector similarity brings the two closer together, but it doesn't guarantee the model will actually be able to use the chunk to answer the question.
The vector says the chunk is relevant. The model still has to be able to read it.
What actually worked in practice
The change that most impacted response quality wasn't in the search. It was in how the content was prepared before indexing.
Two things made a real difference.
The first was enriching the chunk with structural context before generating the embedding. Instead of indexing the paragraph in isolation, I'd prepend the section title, the subheading, and the first few tokens of the previous paragraph. The text going into the embedding function had enough context to represent what the chunk actually meant within the document. The text going to the model was the original chunk, without the prefix.
This partially solves the problem. The vector now represents a contextualized fragment. The search starts retrieving chunks that make more sense.
The second change was generating embeddings from hypothetical questions rather than from the text itself. For each relevant chunk, I'd automatically generate three or four questions that chunk would answer, and index those questions with a reference back to the chunk. At search time, the user's query was compared against hypothetical questions, not affirmative text. Similarity improved in a measurable way.
This has a cost. Slower indexing, more tokens spent, a more complex pipeline. In production, it's worth it if response quality is critical.
The error MCP doesn't fix
After working with MCP in a few contexts, I noticed the problem of poorly generated embeddings resurfaces in a different form when you connect retrieval tools as MCP resources.
MCP solves the boundary between the agent and the external system. It defines when and how the model accesses information. But it doesn't solve what's inside the external system.
If your MCP server exposes a search tool that underneath uses a vector index with bad embeddings, the agent will retrieve bad context more efficiently. MCP will work perfectly. The responses will keep being bad.
The integration layer is right. The representation layer is wrong. And that's hard to debug because everything in the pipeline appears to be working.
Honest trade-offs
Enriching chunks with structural context increases the size of the indexed documents. Depending on volume, this affects storage cost and search latency.
Generating hypothetical questions significantly increases indexing cost. On a large corpus, this can be prohibitive without an incremental indexing strategy.
And there's a subtler problem: the context you add to the chunk to improve the embedding can introduce noise if the document is poorly structured. Inconsistent documentation, generic headings, sections with no clear hierarchy. In those cases, the context prefix might make the embedding represent something wrong with even more confidence.
What I'd do differently today
I'd start by evaluating embedding quality before building anything on the search side.
Take a set of real questions the system should be able to answer. Retrieve the chunks. Manually assess whether the retrieved chunks are the ones you'd expect. If precision is bad at this stage, no downstream adjustment will fix it.
And I'd swap out the success metric. Instead of measuring search precision in isolation, I'd measure final response quality by question type. That's the only metric that matters in production.
The counterintuitive part of RAG systems is that the most invisible step, generating the embedding, is the one that most determines the quality of the entire system. You can have the best model available, a well-calibrated search pipeline, a correct MCP server, and still deliver bad responses if the vector doesn't represent what the model will actually need to read.
The problem doesn't show up in the logs. It shows up in the response the user gets.