Advanced enterprise RAG patterns: what separates a demo from a system that holds

Naive RAG gives a convincing demonstration and a disappointing system, and it never breaks down: it answers, with the wrong chunk, and nobody notices. Here are the five patterns that make the difference, and the order in which to apply them.

By QDNA · Published August 28, 2026 · 11 min read

Why naive RAG disappoints after the demo

The basic scheme works remarkably well on a hundred homogeneous documents, and degrades as soon as you approach a real corpus, for three causes listed here in their observed order of frequency.

Chunking loses the referent, a fragment starting with "this provision does not apply to prior contracts" is unusable: "this provision" referred to something two pages earlier. The fragment then remains syntactically fine and semantically close to the question while being factually useless, which makes it the most frequent failure and the hardest one to spot.

Similarity is not relevance, a vector measures topical proximity. A question about an exception will first find the paragraph stating the rule, because it discusses the same topic more emphatically, and the model will then answer with the rule, confidently.

The corpus contains versions, a 2023 procedure, its 2025 revision and a draft that was never approved look very similar in vector space. Without a version field, the system answers at random among the three : and the user has no way to tell.

These three failures share one property, which explains why they survive so long in production: the system always answers, since instead of returning an error it returns a plausible wrong answer that nothing flags.

Pattern 1 : Chunk while keeping context

Fixed-size chunking is the default in every library, and the first thing to replace.

Three measures, which compound without contradicting one another, address the lost referent introduced at chunking.

• Chunk on structure rather than on a character count: by section, by article, by table cell. A document already has semantic boundaries; they are in its outline.
• Prefix each fragment with its path : document title, chapter, section. The fragment above becomes "Collective agreement X › Article 12 › Scope: this provision does not apply…". The cost is a few dozen tokens per fragment; the gain is that it becomes interpretable on its own.
• Also index isolated sentences when the corpus contains point facts. A fact stated as an aside inside a paragraph about something else is drowned if you only index blocks: the block vector is dominated by the main topic, not by the aside.

These three measures are not a matter of intuition, and their effect has been quantified. Anthropic published the gain obtained by prefixing each chunk with its context before embedding it, measured against a retrieval failure rate within the top twenty chunks: contextual embeddings alone bring that rate from 5.7% down to 3.7%, a 35% reduction, and combining them with a contextual BM25 index brings it to 2.9%, a 49% reduction. The added context represents 50 to 100 tokens per chunk, at a one-off generation cost of $1.02 per million document tokens.

Source: Anthropic, Contextual Retrieval. Those rates were measured on their evaluation set rather than on your corpus, which is precisely what the fifth pattern is about.

Pattern 2 : Filter by clearance before scoring

This is the most important pattern for an enterprise deployment, and the one tutorials omit because it does not exist in a demo dataset.

The temptation is to retrieve the best fragments, then drop the ones the user is not allowed to see, which amounts to a design flaw on two separate counts. First, the number of remaining fragments becomes unpredictable: if the top eight are filtered out, nothing is left to answer with. Second, and more importantly, system latency varies with the content dropped, which is enough to reveal the existence of documents to someone without access to them.

The clearance filter must therefore apply inside the index query, as a constraint, before similarity is computed. In practice that means every fragment carries the access metadata of its source document, and that this metadata is updated when rights change : a point most projects discover late.

The corollary is covered in our article on sensitive data: deleting a source document does not purge its vectors, and a revoked right does not propagate to the index by itself either.

Pattern 3 : Decompose multi-hop questions

"What is the notice period for a manager hired after the 2025 revision?" asks for three facts that coexist in no single paragraph: the revision date, the manager status and the notice schedule, so that a single search cannot bring them back together.

The pattern is to have the model produce, before any search, the list of sub-questions to resolve, then run one search per sub-question and assemble. It is slower and more expensive, which means enabling it only on questions that deserve it : an upstream classifier, like the one described in the orchestration engine, is enough to make that call.

A useful warning: systematically rewriting the question before searching is often presented as an automatic gain. It is not, and assuming otherwise will cost you measurable recall on your own corpus. On a corpus whose vocabulary is very close to the users', rewriting the question moves it away from the documents. Query expansion techniques must be measured on your corpus before adoption; several of them degrade recall in settings where the literature reports them as wins.

Pattern 4 : Retrieve small, return large

There is a tension between the system's two needs: search is more precise on small fragments, generation is better with extended context.

The pattern that resolves it separates the indexed unit from the transmitted unit. You index short, highly discriminating fragments; when one of them surfaces, you pass the model the parent block it belongs to : the whole section, the short document, the neighbouring paragraphs.

The third row of the table is anything but cosmetic, because a system that cites verifiable sources changes the very nature of the error: the user sees that the excerpt does not say what the answer claims, whereas without citation the same error stays undetectable.

The same measurement quantifies what reranking adds: applied on top of contextual embeddings and BM25, it brings the failure rate from 5.7% down to 1.9%, a 67% reduction. The source also notes that passing the top twenty chunks outperforms passing only the top five or ten, which runs against the instinct to economise context.

On separating the indexed unit from the transmitted unit, see also H-RAG, SemEval-2026 Task 8, which describes a hierarchical parent-child pipeline separating fine-grained retrieval from the context passed on.

Pattern 5 : Measure recall before quality

This is the pattern that governs the other four, and the one most often missing.

The temptation is to evaluate end to end, asking questions and judging answers, but the problem is that this single number aggregates two independent causes : the right passage was not retrieved, or it was retrieved and poorly used. The two failures call for opposite fixes, and a global score does not tell you which one you have.

Measurement that actually informs a decision happens in two distinct steps, whose order matters.

1. Retrieval recall: on a set of questions whose expected passage you know, how often is that passage among the retrieved fragments? It is an objective figure, independent of any generation model, and it bounds everything else. If recall is 60%, no prompt improvement will beat 60%.
2. Answer quality given the passage: by deliberately supplying the right passage, does the model answer correctly? If yes, your problem is retrieval; if not, it is generation or context formatting.

Building the annotated question set is the thankless work that makes everything else possible. A hundred questions with their expected passage are enough to tell a real improvement from an impression, which no demonstration can do.

What these patterns do not fix

Three limits are worth stating, because they often lead teams to stack patterns where the problem lies elsewhere.

A contradictory corpus stays contradictory, if three valid documents give three different answers, no retrieval technique will arbitrate. The system should then flag the disagreement rather than pick : behaviour that has to be coded, and therefore decided.

Aggregation questions are not a RAG problem, since a question such as "how many contracts expire this quarter?" asks for a count over the whole corpus rather than the retrieval of a few excerpts, which makes it a query over structured data: extracting the information into a database and querying that costs less and returns a correct result.

RAG does not replace document governance, indexing a poorly maintained corpus produces a system that answers authoritatively from obsolete documents, which makes corpus quality the dominant variable, well ahead of any technical refinement.

Where to start

The implementation order that avoids dead ends starts with measurement rather than with patterns, contrary to intuition.

1. Build fifty to a hundred annotated questions with their expected passage. Without that set, everything that follows is intuition.
2. Measure the current system's recall, the number is almost always lower than the team estimated.
3. Fix chunking : the best effort-to-gain lever, and the one that most often explains mediocre recall.
4. Put the clearance filter in the query before adding anything else. Retrofitting it later means reindexing.
5. Add decomposition and parent blocks only if measurement shows they help on your corpus.

The general pipeline frame, from chunking to hybrid search, is set out in our enterprise RAG guide. To serve the whole thing on your own infrastructure, see vLLM and the sizing matrix, or a conversation to frame the first milestone.