> ## Content Index
> Fetch the complete content index at: https://vanta.planethemes.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Context Engineering: Getting Past the RAG Plateau
- URL: https://vanta.planethemes.com/context-engineering-rag-plateau/
- Published: 2026-06-29T09:00:00.000Z
- Updated: 2026-08-20T13:46:08.000Z
- Description: Retrieval got us to demos. Context engineering — budgets, compaction, and structured memory — is what gets us to production.
- Author: Elena Rostova
- Tags: Artificial Intelligence, #Import 2026-08-20 13:46

Most teams discover the same thing three months into their AI feature: retrieval quality was never the bottleneck. Context assembly was. The embeddings are fine. The vector database is fine. What is broken is everything that happens between a retrieval call returning and a prompt hitting the model.

## The plateau, diagnosed

Naive RAG pipelines stuff every marginally relevant chunk into the prompt and hope the model sorts it out. Past a certain scale, adding retrieval recall makes answers worse, not better — the signal drowns in near-duplicates, stale versions of the same document, and passages that are topically adjacent but factually irrelevant. Models degrade measurably when forced to hunt for one relevant sentence inside twenty plausible-looking paragraphs.

The teams that break through the plateau stop asking "how do we retrieve better chunks" and start asking "what does the model actually need to see to answer this class of question." Those are different engineering problems with different owners, different metrics, and different fixes.

## Budgets, not buffers

Production systems treat the context window as a budget with line items: system instructions, conversation memory, retrieved evidence, and working space each get an explicit allocation, and something is accountable for every token spent. When the evidence allocation is 4,000 tokens, the retrieval layer must rank, deduplicate, and compress to fit — which forces the quality conversation that unlimited stuffing lets you avoid.

A useful smell test: if nobody on the team can say what fraction of the prompt is instructions versus evidence on a typical request, the context is a buffer, not a budget, and quality is drifting with every feature added upstream.

## Compaction is a feature, not a hack

Structured summaries beat raw chunks almost everywhere. A retrieval layer that rewrites five overlapping passages into one attributed summary — with document IDs preserved for citation — routinely outperforms shipping the raw text, because the model spends its capacity reasoning instead of reconciling. The same applies to conversation history: recency-weighted compaction, where old turns collapse into a running summary, beats both infinite history and brutal truncation.

## The query is half the battle

A small, fast model that rewrites the user query before retrieval — expanding acronyms, resolving pronouns against the conversation, splitting compound questions — improves end-to-end quality more cheaply than upgrading the answering model. Retrieval is only as good as the question it receives, and users do not write questions for your retriever; they write them for a colleague.

## Evaluate the assembly, not just the answer

End-to-end evals hide where failures happen. Mature pipelines score stages independently: did retrieval surface a passage containing the answer, did compaction preserve it, did the model use it. Instrumenting those three questions turns "the AI is being weird" into a bug report with an owner — and that, more than any model upgrade, is what gets you past the plateau.