Pedro Sousa
‹ ← Articles

Article · Blog

The document you indexed is not the document the model needs to read

RAG fails silently when the ingestion pipeline treats every document as if they were the same. The problem is not in the vector search. It's in what you decided to index and how you structured it before any embedding.

The document you indexed is not the document the model needs to read

When I put together the first real RAG pipeline for a production system, recall was good. The search was returning the right chunks. The model had access to the correct information.

And yet, the answers were wrong.

Not drastically wrong. Wrong in a subtle way: the model was using the information, but without enough context to apply it well. It knew what was written. It didn't know what that meant inside the system.

It took me a while to figure out where the problem was.


The pipeline that looked right

The architecture was the one every tutorial shows: you take the documents, split them into chunks, generate embeddings, store them in a vector database, and at query time you do a similarity search and inject the results into the context.

It works on paper. It works in demos. It fails in production in a specific way that's hard to diagnose because the individual components are all operational.

The problem lies in an implicit assumption you probably accepted without questioning: that the document that exists in your system is the document the model needs to read.

It's not.


What the original document carries

Internal documents from a real system carry a lot of implicit context.

A technical specification written for engineers assumes that whoever reads it knows the project's history, the decisions that came before, the internal acronyms, what was discarded and why. A PDF of a usage policy written by legal assumes the reader knows the underlying regulation. A changelog assumes the reader followed the previous version.

When you slice that document into chunks and index them, you lose much of that implicit context. The chunk returned by the vector search exists in a vacuum. It has the right words. It doesn't have the context that makes those words understandable.

The model reads the chunk. It doesn't read what was in the head of whoever wrote the document.


The problem most people ignore: ingestion is modeling

There is a step that most RAG implementations treat as operational when they should treat it as architectural: the transformation of the document before embedding.

I'm not talking about chunking strategy. Chunk size, overlap, separators... all of that matters. But it matters less than the prior decision: what you decide the chunk needs to contain in order to be useful in isolation.

In a support system I built, the problem showed up like this: the search was returning the right chunk about a business rule, but the chunk didn't say in which context that rule applied. The model would generate a technically correct answer but applied to the wrong case.

The solution wasn't to improve the search. It was to change what the chunk contained.

Each chunk started including explicit metadata in the text itself: the scope of the source document, the type of content, the preconditions that applied to that section. Not just as filter metadata in the vector store. Inside the text the model would actually read.

This is what I call ingestion contextualization: treating the document preparation step as domain modeling, not as a technical transformation.


Where MCP changes the problem

Model Context Protocol solves a different, but related, problem.

RAG assumes you can anticipate what the model will need and pre-index it. For many cases, that assumption holds. For dynamic systems, it breaks.

When the model needs information that changes constantly, or needs to navigate a structure that depends on the current state of the system, static RAG is not the right tool. You indexed a snapshot. The system has moved on.

MCP solves this by exposing tools the model can call in real time. Instead of searching a static index, the model queries the system directly when it needs to.

The mistake I've been seeing is treating MCP as a replacement for RAG. It's not. They are different layers.

RAG is long-term memory over stable knowledge. MCP is operational access to dynamic systems. Mixing up the use cases is what generates confused pipelines where the model sometimes searches the index, sometimes calls a tool, with no clear criteria, and the answers end up inconsistent depending on which path got triggered.


The decision most people never make

What most implementations lack is not technology. It's an explicit decision about what type of knowledge each mechanism should serve.

Stable domain knowledge, documentation, policies, specifications that rarely change: RAG, with careful ingestion and explicit contextualization of chunks.

Current system state, user data, information that changes during the session: tools via MCP or function calling, real-time calls.

Knowledge the model needs in order to reason well about the domain before any query: system prompt, not RAG, not MCP.

When you don't make that decision, the system tries to solve everything with a single mechanism. And it fails at edge cases constantly, because each mechanism has different characteristics around latency, coherence, and freshness.


What I would do differently

In the first pipeline I built, I treated ingestion as infrastructure. I picked the chunker, configured the size, ran the embedding, pushed it to the database. Done.

Today I treat ingestion as the most important step in the RAG pipeline, not the most mechanical one.

Before indexing any document, the question I ask is: if I take this chunk in isolation and show it to someone who doesn't know the system, will they understand what this piece means and where it applies? If the answer is no, the chunk is not ready to be indexed.

That implies real transformation before embedding: adding explicit context, normalizing internal terminology, disambiguating references that depend on implicit knowledge, sometimes rewriting entire sections so they make sense outside the original document.

It's work. It's more work than configuring chunking parameters.

And it's the difference between a RAG pipeline that works in a demo and one that works in production.


The illusion of high recall

There is a metric that is very misleading when developing RAG systems: search recall.

You measure whether the right chunks are being returned. Recall is at 90%. Looks good.

But recall measures whether the correct chunk was retrieved. It does not measure whether the chunk contained enough information for the model to answer well. Those are metrics for different problems.

You can have perfect recall and poor response quality if the right chunk was poorly contextualized.

The problem wasn't in your vector database. It was in what you decided to put inside it.

That's the counterintuitive part: optimizing embeddings and search is the visible work. Preparing documents so they are understandable outside their original context is the invisible work that determines whether the system actually functions.

Most people invest in the first and underestimate the second.