Article · Blog
The model didn't forget. You never explained it properly.
Most problems with LLMs in production aren't model failures. They're failures by whoever built the system around it. Context engineering isn't an optional step: it's the only part that actually matters.
The model didn't forget. You never explained it properly.
In a financial app with millions of users, we integrated an agent to answer questions about account transactions. The model was good. Tests were promising. In production, the agent started giving vague answers, sometimes contradictory, sometimes correct but useless.
The initial hypothesis was "the model needs fine-tuning." We spent two weeks going down that road.
The real problem was something else: the agent didn't know who the user was, what stage of the journey they were at, or what had happened in the last few interactions. The model was receiving a question with zero operational context and trying to be helpful with what it had. Sometimes it got lucky.
Fine-tuning doesn't fix that. No model fixes that.
The mistake most people make without realizing it
There's an implicit belief that a good enough LLM will compensate for the lack of context. That the model will "infer" what's missing, "deduce" the intent, "fill in" the gaps.
It won't.
An LLM is a sequence-completion machine based on what it received. If what it received is ambiguous, incomplete, or disconnected from the system's reality, the output will reflect exactly that.
The problem isn't in the model. It's in what arrives before the model.
I call this the Illusion of Implicit Intelligence: the assumption that the model knows things nobody ever told it.
Context is not the system prompt
When most people talk about "context," they think of the system prompt. Define the agent's role, add some instructions, maybe one or two examples.
That's the bare minimum. And the bare minimum is rarely enough in production.
Real context is a set of layers:
Operational context: who the user is right now, what they're trying to do, which step of the flow they're at, what happened earlier in this session.
Historical context: what this user did in their last interactions, what issues they reported, what their behavioral pattern looks like in the system.
State context: the current state of the system, the relevant entities, the data the agent needs to make an informed decision.
Constraint context: what the agent can and cannot do, which actions have irreversible consequences, where it should stop and escalate to a human.
When any of these layers is missing, the agent starts improvising. And an agent improvising in production is exactly what it sounds like.
The problem of context that grows and ages
There's a second problem that shows up after you solve the first one.
You build rich context. The agent starts working well. Then the interaction history grows, user data accumulates, the state system gets more complex. The context that used to fit comfortably in the model's window is now competing with everything else.
And you start cutting pieces to fit within the token limit.
Here's the decision I've seen go wrong most often: cutting context linearly, removing the oldest interactions. Seems reasonable. In practice, you sometimes remove exactly the piece of information that explains the current state.
The user mentioned in the third message that they were having trouble with a specific payment. The tenth message only makes sense with that history. You cut the third one to save tokens. The agent lost the thread.
Context isn't just volume. It's relevance. And relevance is hard to calculate generically.
What worked best for me was to stop thinking in terms of "context window" and start thinking in terms of "intentional working memory." What's the information without which this agent cannot solve this problem right now? That stays. Everything else is negotiable.
Tool calling is not a shortcut for context
Another pattern I've seen fail a few times: using tools (function calling / tool use) as a substitute for context.
The logic is: the agent will fetch whatever it needs when it needs it. You don't have to put everything in the initial context.
This works for data that genuinely doesn't make sense to load upfront. But for information that defines who the user is, what they want, and what has already happened in this conversation... it doesn't work.
Every tool call is latency. It's a potential point of failure. It's a coupling point with an external system.
And more than that: when the agent needs to call a tool just to understand the basic context of the conversation, that's a signal the system was designed outside-in. Think of the agent as the center and build the infrastructure around it. In practice, the agent was dropped on top of an existing system and is now trying to reconstruct context the system never surfaced for it.
The agent should arrive at the first message already knowing who the user is, what state they're in, and what's relevant right now. Not discover that mid-conversation.
What changed in my approach
I started treating context engineering as a discipline separate from agent engineering.
They're two distinct problems.
Agent engineering: what tools it has, how it makes decisions, how it handles failures, how it escalates to humans when needed.
Context engineering: what reaches it before any decision is made, how that content is structured, how it stays relevant throughout the conversation, how it ages in a controlled way.
When the two are mixed together, the system becomes hard to debug. You can't tell whether the agent made a bad decision because its logic is flawed or because it didn't have the right information.
Separating the two layers changes how you diagnose. And it changes how you evolve the system.
An honest trade-off
Rich context costs. It costs tokens, it costs latency, it costs complexity in the system that prepares that context.
There's a point where the cost of assembling the perfect context exceeds the cost of letting the agent occasionally be wrong and handling the error some other way.
That's real. And it will depend on the domain. In an agent making financial decisions, the cost of an error is high enough to justify more expensive context. In an agent suggesting content, maybe not.
What I don't recommend is making that decision by default. Most teams cut context because it's simpler to implement, not because they actually analyzed the trade-off.
What stays
A bad model with good context beats a good model with bad context. Every time.
That goes against the instinct of people who enter the problem wanting to pick the best available model. The model matters. But it's the last factor in the chain.
The system you build around the model is what determines whether that model will behave as expected in production.
And that system starts long before the prompt.