You’ve Mastered Prompting. Now What? Meet Context Engineering

If you’ve been following the 5C framework, you already know how to write a clear and well-structured prompt. Clarity, context, constraints, calibration, and character- these 5Cs will make the output of a single AI call better. But then you go build something bigger. An agent that runs for ten turns, reads a few files, calls a couple of tools, and keeps a conversation going. And somewhere around turn six, it starts forgetting things it knew at turn two. It repeats a question you already answered. It picks the wrong tool even though the right one was described right there in its own instructions.

What Is Context Engineering?

Anthropic’s applied AI team, in a recent post, named this context engineering. Prompt engineering is about how you write instructions. Context engineering is about everything else the model sees when it generates a response: the system prompt, but also tool definitions, retrieved documents, conversation history, and whatever the agent has written to memory. Prompt engineering is a subtask inside a much bigger task.

A perfectly written prompt sitting behind three thousand tokens of stale chat history and a pile of irrelevant tool outputs is technically a perfectly written prompt. The model may just never really “see” it the way you intended because it’s buried in the clumsy context.

Why the Model Starts Losing the Thread

LLMs don’t have infinite, uniform attention across their context window. Anthropic calls this phenomenon  context rot: as the number of tokens in context grows, the model’s ability to accurately recall any specific piece of it goes down. It is because every token competes for the same limited “attention budget.”

The architectural reason for this is that the transformer models allow every token to attend to every other token using the attention mechanism.  This roughly makes  relationships for  tokens making relevant information compete with irrelevant information. Also, the models simply see far more short sequences than long ones during training. So, these models suffer significantly when reasoning over genuinely long, cluttered contexts. This results in a gradual decline in precision. So, the fix is not “adding more context” but exactly the opposite.

The Anatomy of Good Context

Anthropic frames the goal simply as “find the smallest set of high-signal tokens that maximizes the chance of the outcome you want”.  For example, if an LLM is helping to debug a production issue, providing the entire repository, months of logs, and every previous conversation is not suitable. Instead, providing the failing code, the stack trace, the error logs, and the recent code changes will enable the model to give usable outputs. The model doesn’t need more context. It needs the right context.

The next obvious question is “Where does this context come from?”

In an LLM application, the context window is made from several components. Below are the important components of the context window.

System prompts

These prompts should operate at a high level of abstraction. If it is too rigid with hardcoded if-else logic trying to cover every case, then the prompt becomes brittle the moment reality doesn’t match your assumptions. If the prompt is too vague, the model has nothing concrete to act on. The sweet spot is specific enough to guide behavior but flexible enough for the model to exercise its judgment.

Tool Definitions

This is where many agents quietly fail. If a tool’s definition is ambiguous, or two tools overlap in what they do, the agent will guess wrong at exactly the moment it matters. A simple, useful rule is:  if a human cannot confidently decide which tool to use by reading the definitions, an LLM is unlikely to make the right choice consistently.

Retrieved and historical context

Retrieved documents and conversation history should never be inserted into the context window without necessary filtering. Every unnecessary token competes for the model’s attention. Poor retrieval, irrelevant documents, or badly chunked content reduce the output quality. The goal is to retrieve relevant information, rank it well, and include only the necessary context for the current task.

Handling Long-Running Agents

For tasks that span many turns, such as a large code refactoring or a multi-step research job, they eventually exceed the model’s context window. Instead of continuously expanding the context, Anthropic outlines three techniques worth knowing.

1. Compaction summarizes a conversation as it nears the context limit and reinitializes with that summary, keeping the important decisions and dropping redundant tool output.
2. Structured notetaking enables writing persistent notes outside the context window, a to-do list, or a running notes file, that get pulled back in later when required.
3. Sub-agent architectures hand off deep, exploratory work to a specialized sub-agent that might use tens of thousands of tokens internally, then reports back a condensed summary to the main agent.

Visual Representation of Context Engineering Vs Prompt Engineering

Fig 1: Prompt engineering vs context engineering in AI systems.

Where This Leaves You

If you’ve read LLMOps on what happens after a GenAI app goes live, this will sound familiar. Building a working prototype is largely a prompting problem. Keeping an agent reliable across hundreds of real conversations, day after day, is a context problem. Same underlying shift, just showing up at a different stage of the pipeline.

You don’t need to solve all of this before you ship anything. Pick one agent you already have running, and for a week, log everything that goes into its context on a single turn: the system prompt, every tool result, every retrieved chunk. You’ll likely find that at least one-third of the data in the context isn’t useful at all.

Then ask a simpler question every time you’re tempted to add something new: does the model actually need this to do the next step well, or is it there just because it was easy to include? This one small habit, applied consistently, will take you further than any framework will on its own.