Article · Blog
RAG didn't fail because of search. It failed because of the context you didn't give the model.
Most RAG problems in production aren't about document retrieval. They're about poorly constructed context passed to the model. This article explains why the pipeline works in testing and breaks in the real world, and what I would change today.
RAG didn't fail because of search. It failed because of the context you didn't give the model.
I put a RAG system into production thinking the hard problem was vector search.
I configured the embedding, tuned the cosine similarity, tested with straightforward questions, the right chunks came back in the top-3. It looked solid. In production, the model started hallucinating in a very specific way: it would blend information from different documents as if they were the same one, and answer confidently about things no chunk ever said explicitly.
The retrieval was correct. The problem was what I was doing with the result.
What most people assume about RAG
There's an implicit idea that RAG solves the model's knowledge problem. You index your documents, search for the most relevant ones, throw them into the prompt, and the model answers based on them.
That idea is right in theory and incomplete in practice.
The model doesn't read chunks the way a human reads a paragraph. It receives a concatenated block of text, with no clear hierarchy, no explicit relationship between the pieces, and tries to infer what matters. When chunks are similar but contradict each other, or when the context is too fragmented, the model fills in the gaps. And it fills them in with confidence.
That's not a bug in the model. It's the expected behavior of a language system when faced with ambiguous context.
The real problem: you're assembling context, not just searching documents
In a system I built for querying technical documentation, the vector search returned the five chunks most similar to the user's question. I concatenated them in score order and passed them to the model.
What I wasn't seeing: two chunks could come from the same document at different versions. Or from documents covering the same topic but reaching opposite conclusions. With no structural anchor in the prompt saying "these are three distinct sources," the model treated them as a coherent context.
The failure didn't show up on simple questions. It showed up when users asked questions that crossed concepts, when the right answer depended on understanding that multiple sources existed, when the right tone was "it depends on context X or Y" but the model answered with false certainty.
The retrieval was perfect. The context was garbage.
What changed when I stopped treating context as a string
The first change was structural. Instead of concatenating chunks as plain text, I started assembling context with explicit metadata:
[Source 1 - Document: authentication_v2.md | Relevance: 0.91]
...chunk content...
[Source 2 - Document: authentication_legacy.md | Relevance: 0.87]
...chunk content...
Simple. But the effect was immediate. The model started referencing sources by name, contrasting versions when they differed, and using "according to document X" instead of answering as if there was a single truth.
The second change was in the system prompt. I stopped instructing the model to "answer based on the provided documents" and started instructing it about the nature of the context: "You will receive excerpts from multiple documents that may contain complementary or contradictory information. When there is a conflict, indicate it explicitly."
This isn't sophisticated prompt engineering. It's communicating the structure of the context to the model.
Where MCP fits into this story
When I started working with MCP, the pattern that caught my attention wasn't the ability to call tools. It was the explicit separation between context, tools, and instruction.
MCP forces you to think in layers. What is given as context. What is a tool the model can invoke. What is instruction about how to act.
In a naive RAG system, all of that is mixed into one giant prompt. The model infers the structure. Sometimes it gets it right, sometimes it doesn't.
When I separated those layers, even outside of MCP, the behavior became more predictable. The model knew what was the source of truth, what was the operational instruction, and what was an available tool.
The problem with poorly built RAG is that it treats the model as an upgraded search engine. You retrieve, throw it at the model, wait for the answer. The model became the final step in a pipeline that doesn't think of it as an agent.
The trap of the right chunk in the wrong place
There's a pattern I started calling the Orphan Chunk Problem: retrieval returns a technically relevant excerpt, but one that loses its meaning outside its original document.
A real example: a chunk that said "this behavior was deprecated in version 3.0" was being returned for questions about a specific feature. Alone, without the preceding paragraph that explained which behavior was being referred to, the chunk was useless or confusing. The model tried to infer the antecedent.
The solution wasn't to increase the chunk size. It was to add window context: when retrieving a chunk, automatically include the paragraph before and after it when the relevance score was high but the chunk was small.
It seems obvious once you see the problem. Before you see it, it doesn't.
What test evaluation doesn't capture
Every RAG system seems to work in testing because you test with questions that have direct answers in your documents.
In production, users ask in ways you didn't anticipate. They ask about relationships between concepts. They ask about edge cases. They ask what happens when two scenarios collide.
Those questions stress the quality of the context, not the quality of the search.
It took me longer than it should have to realize that my evaluation suite was testing retrieval and assuming the context was fine. End-to-end evaluation with real users revealed a completely different kind of failure that no similarity metric was capturing.
Trade-offs nobody talks about
Structured context has a cost. More tokens, more latency, more cost per request. When you start including metadata, window context, and source separation, the prompt grows.
There's a point where adding too much context degrades quality because the model loses track of the most relevant chunks in all the noise. The problem doesn't go away, it inverts.
The answer I found was active prioritization: instead of always passing the top-5 chunks, pass the top-3 with structured context. Less volume, more structural clarity. For my use case, that was the combination that worked.
It's not a universal answer. It depends on the domain, the document size, the model.
What I would do differently today
I would start by evaluating context quality before retrieval quality.
Retrieval is easier to measure because you have established metrics: precision@k, recall, MRR. Context quality is harder to measure and easier to ignore.
But it's where most systems fail.
Perfect retrieval with poorly assembled context produces confident and wrong answers. Which is the worst kind of failure in a system the user is going to trust.
The model won't warn you that the context is ambiguous. It will answer anyway.
That's the difference between a RAG system that works in a demo and one you actually trust in production.