RAG Architecture Patterns

RAG (Retrieval-Augmented Generation) has evolved past the textbook “retrieve top-K chunks, stuff in prompt” picture. Production RAG is a family of architectures with different trade-offs. Knowing the patterns is what differentiates “I’ve used a RAG tutorial” from “I’ve shipped RAG in production.”

For RAG basics see What is RAG?. For embeddings see What are embeddings. For vector DBs see Vector Databases. For chunking/retrieval see Chunking.

The four evolutionary stages

Stage	What
Naive RAG	retrieve → stuff → generate. The tutorial pattern.
Advanced RAG	adds pre-retrieval and post-retrieval optimization (query rewriting, reranking, filtering).
Modular RAG	reconfigurable pipeline of components (retrievers, fusion modules, scorers).
Agentic RAG	LLM decides what to retrieve and when; multi-step / iterative retrieval.

Most production systems are at “advanced” with elements of modular. Agentic is the frontier.

Naive RAG (the baseline)

text

Question → Embed → Vector search → Top-K chunks → Stuff in prompt → LLM → Answer

python

def naive_rag(question: str, top_k: int = 5) -> str:
    query_embedding = embed_model.encode(question)
    chunks = vector_db.search(query_embedding, top_k=top_k)
    context = "\n\n".join(c.text for c in chunks)
    prompt = f"Context:\n{context}\n\nQuestion: {question}\n\nAnswer:"
    return llm.generate(prompt)

What’s wrong:

Query-document mismatch: questions and documents have different distributions. “How do I reset my password?” doesn’t embed close to “Click Settings → Account → Security → Change Password.”
Top-K is blunt: low K misses; high K dilutes the prompt with noise.
No filtering: retrieved irrelevant chunks confuse the LLM.
No reranking: vector similarity is approximate; the most relevant chunk isn’t always the highest-scored.
Context window limits: stuffing K=20 chunks may overflow or waste tokens.
No quality signal: you don’t know if retrieval failed; LLM may hallucinate.

Naive RAG is fine for demos. It’s not production.

Advanced RAG — pre and post retrieval optimization

text

                  ┌─ pre-retrieval ─┐                      ┌─ post-retrieval ─┐
Question → [query rewrite, decomposition] → Search → [rerank, filter, compress] → Stuff → LLM → Answer

Pre-retrieval techniques

Query expansion / HyDE (Hypothetical Document Embeddings): LLM generates a hypothetical answer; embed the hypothetical answer (not the question); search.

python

def hyde(question):
    hypo = llm.generate(f"Write a concise paragraph answering: {question}")
    hypo_embedding = embed_model.encode(hypo)
    return vector_db.search(hypo_embedding, top_k=5)

Question and document distributions get closer. Works well for FAQ-like queries.

Query decomposition: break complex questions into sub-questions.

text

"Compare Python and Go performance for ML workloads."
→ ["Python performance for ML",
   "Go performance for ML",
   "Python vs Go performance comparison"]

Retrieve for each; merge.

Step-back prompting and query routing

Step-back prompting: ask the model for the more general question behind the specific one, retrieve for both, and merge.

text

"Did the 2024 refund policy apply to annual plans bought in December?"
→ step-back: "What is the refund policy for annual plans?"

The specific question often matches no chunk, because no document is written at that level of detail. The general one matches the policy page that contains the answer. Useful for reasoning-heavy questions over policy, legal and documentation corpora; wasted effort on lookup questions that already match.

Query routing: classify the question, route to the appropriate index / strategy.

python

def route(question):
    category = classifier.predict(question)
    if category == "technical":
        return docs_index.search(question)
    elif category == "billing":
        return billing_kb.search(question)
    # ...

Post-retrieval techniques

Reranking: vector search gives 50 candidates; a cross-encoder (more expensive but more accurate) reranks down to top 5.

python

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rag_with_rerank(question, top_k=5):
    candidates = vector_db.search(embed(question), top_k=50)
    pairs = [(question, c.text) for c in candidates]
    scores = reranker.predict(pairs)
    ranked = [c for c, s in sorted(zip(candidates, scores), key=lambda x: -x[1])]
    return ranked[:top_k]

Vector search is fast but uses bi-encoder embeddings (independent query/doc encoding). Cross-encoders look at the pair together; much more accurate, much slower. Hence the pattern: fast retrieval → slow rerank.

Filtering: remove obviously irrelevant chunks before sending to LLM. Score thresholds, metadata filters, dedup.

Context compression: summarize or extract only relevant parts of long chunks.

python

def compress_context(question, chunks):
    compressed = []
    for chunk in chunks:
        extracted = llm.generate(
            f"Extract only the parts of this text relevant to: {question}\n\n{chunk.text}",
            max_tokens=200,
        )
        if extracted.strip():
            compressed.append(extracted)
    return compressed

Trades latency (one LLM call per chunk) for cleaner context. Use sparingly.

Modular RAG

Treat RAG as a directed graph of swappable modules:

text

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│ Query Aug   │ →  │ Retriever   │ →  │ Reranker    │
└─────────────┘    └─────────────┘    └─────────────┘
                          │                  │
                          ↓                  ↓
                   ┌─────────────┐    ┌─────────────┐
                   │ BM25 Index  │    │ Generator   │
                   └─────────────┘    └─────────────┘

Each module is swappable. Hybrid retrieval combines dense (vector) and sparse (BM25); the fusion module merges scores. Re-rankers slot in or out. Pre-retrieval augmentation routes through different paths.

Frameworks: LlamaIndex, Haystack, DSPy. They expose this as a pipeline configuration.

python

# LlamaIndex example
from llama_index.core import VectorStoreIndex
from llama_index.core.retrievers import VectorIndexRetriever, BM25Retriever
from llama_index.core.query_engine import RetrieverQueryEngine

vector_retriever = VectorIndexRetriever(index=vector_index, similarity_top_k=10)
bm25_retriever = BM25Retriever.from_defaults(docstore=docstore, similarity_top_k=10)

fused_retriever = HybridRetriever(vector_retriever, bm25_retriever)
query_engine = RetrieverQueryEngine.from_args(fused_retriever, reranker=rerank_module)

Agentic RAG

LLM decides what to retrieve and when, often iteratively. The model is the planner.

text

LLM: "I need to look up X first."
→ Retrieves X
→ LLM: "Based on X, I also need Y."
→ Retrieves Y
→ LLM: "Now I have enough — final answer."

Built on tool-calling: the LLM emits a “search” tool call; the agent executes; result fed back; LLM continues.

python

def agentic_rag(question):
    messages = [
        {"role": "system", "content": "You can search the knowledge base. Use it when needed."},
        {"role": "user", "content": question},
    ]
    while True:
        response = llm.chat(messages, tools=[search_tool])
        if response.tool_calls:
            for call in response.tool_calls:
                result = vector_db.search(call.args["query"])
                messages.append({"role": "tool", "content": result, "tool_call_id": call.id})
        else:
            return response.content

Trade-offs

Pros:

Multi-hop reasoning (“first find X, then use X to find Y”).
Adaptive: simple questions skip retrieval entirely; complex ones iterate.
Self-correcting: LLM can recognize bad retrieval and search again.

Cons:

Latency: each iteration is a round trip.
Cost: many LLM calls + many retrievals.
Failure modes: infinite loops, getting confused, deciding not to search when it should.

For complex reasoning use cases (research, analysis, coding agents): agentic RAG is the right tool. For simple Q&A: advanced RAG suffices.

See The agent loop.

Specialized patterns

Self-RAG

LLM evaluates its own retrievals: “Is this relevant? Is my answer grounded?” Built-in self-correction.

text

Retrieve → LLM: "Are these chunks relevant?" → if no, retrieve again
LLM generates → LLM: "Is my answer supported by the chunks?" → if no, regenerate

Reduces hallucination but adds 2-3× LLM calls.

Corrective RAG (CRAG)

A retrieval evaluator scores chunks. If low confidence, fall back to web search or a different index.

python

def crag(question):
    chunks = vector_db.search(embed(question))
    scores = evaluator.predict(question, chunks)
    if max(scores) < THRESHOLD:
        chunks = web_search(question)
    return generate(question, chunks)

Multi-vector retrieval

Store multiple embeddings per document:

Summary embedding (for high-level matching).
Hypothetical question embeddings (for FAQ-like queries).
Chunk embeddings (for fine-grained retrieval).

Different retrieval strategies hit different embedding indices.

Parent-document retrieval

Embed small chunks (precise matching) but return their larger parent documents (full context):

text

Embed: each paragraph
Retrieve: top-K paragraphs by similarity
Return: the parent documents (or sections) of those paragraphs

Best of both worlds: precise matching + complete context.

Graph-augmented RAG

For knowledge graph data, retrieve related entities and their relationships, not just text chunks.

text

Question: "Who manages projects that use Python?"
→ Identify entities (people, projects, technologies)
→ Traverse graph (people MANAGES project USES python)
→ Synthesize result

Used for compliance, recommendation systems, structured Q&A.

Production architecture (typical stack)

text

User question
    ↓
┌─────────────────┐
│ Query rewriting │ → optional HyDE / decomposition
└─────────────────┘
    ↓
┌─────────────────┐
│ Hybrid retrieval│ → dense (vector) + sparse (BM25), fuse with RRF
│  - vector store │
│  - BM25 index   │
└─────────────────┘
    ↓
┌─────────────────┐
│ Reranker        │ → cross-encoder, top 50 → top 5
└─────────────────┘
    ↓
┌─────────────────┐
│ Filter / dedup  │ → score threshold, metadata filters
└─────────────────┘
    ↓
┌─────────────────┐
│ Compress        │ → only if context too long
└─────────────────┘
    ↓
┌─────────────────┐
│ LLM             │ → prompt with retrieved context + instructions
└─────────────────┘
    ↓
┌─────────────────┐
│ Citation check  │ → optional: verify claims trace to chunks
└─────────────────┘
    ↓
Answer + sources

Most production RAG systems are some subset of this. Start naive, add components as needed.

Failure modes

Failure	Cause	Mitigation
Hallucinated answers	Retrieved chunks don’t contain the answer	Better retrieval; LLM grounding instructions; cite sources
Lost-in-the-middle	LLM ignores middle chunks in long context	Reranking; reorder so most relevant chunks are first/last
Outdated information	Stale index	Periodic re-indexing; freshness scoring
Wrong domain	Index doesn’t have the topic	Routing / fallback to web search (CRAG)
Empty / sparse results	Niche query	Query expansion; sparse retrieval (BM25) backup
Off-topic chunks	High vector similarity for irrelevant text	Reranking; threshold; metadata filters
Repeated info	Many chunks say the same thing	Deduplication; diversification

Cost vs quality knobs

You’re always trading these:

Knob	More quality	More cost
Number of retrieved chunks	higher	bigger prompt, more $$
Reranking	better top-K	extra latency + compute
HyDE	better recall	one extra LLM call per query
Iterative agentic	multi-hop reasoning	N× latency + cost
Self-RAG	grounded answers	2-3× LLM calls
Cross-encoder reranker	better quality	slower than rerank-light models
Larger embedding model	better semantic match	slower indexing, more storage

Real systems benchmark each knob against held-out queries; tune for the cost/quality target.

Common interview confusions

“RAG and fine-tuning are alternatives.” — not quite. RAG injects facts; fine-tuning teaches style or behavior. Often combined: fine-tune for tone, RAG for knowledge.
“More chunks = better.” — past ~10 chunks, LLMs start ignoring middle content. Reranking + small top-K beats stuffing 50 chunks.
“Vector search is all you need.” — pure dense retrieval misses keyword-heavy queries. Hybrid (dense + BM25) catches both.

Interview angle 8

“What are the RAG architecture patterns?” — four evolutionary stages: naive (retrieve-then-generate), advanced (pre/post retrieval optimization), modular (swappable components in a pipeline), agentic (LLM decides when/what to retrieve, iteratively).
“What’s wrong with naive RAG and how do you fix it?” — query-document mismatch (HyDE), bad ranking (rerankers), noise in context (filtering + compression), missing recall (hybrid retrieval), no quality signal (eval + grounding checks).
“What’s HyDE?” — Hypothetical Document Embeddings. LLM generates a hypothetical answer to the question; you embed and search using the hypothetical answer. Question and document distributions get closer; better retrieval.
“What’s reranking and why use it?” — fast vector search returns 50+ candidates; an expensive cross-encoder rescores them; top 5 go to the LLM. Cross-encoders see the (query, document) pair together — much more accurate than independent embeddings.
“What’s agentic RAG?” — LLM uses retrieval as a tool, calling it iteratively when needed. Supports multi-hop reasoning, adaptive retrieval (skip if not needed), self-correction. Slower and more expensive but handles complex queries.
“What’s the ‘lost in the middle’ problem?” — LLMs disproportionately ignore information in the middle of long contexts. Mitigate by reranking and placing the most important chunks at the start or end of the prompt.
“Hybrid retrieval — what and why?” — combine dense (vector) and sparse (BM25/keyword) retrieval. Dense catches semantic similarity; sparse catches exact terms. Fuse with Reciprocal Rank Fusion (RRF). Better recall than either alone.
“How do you reduce hallucinations in RAG?” — instruct LLM to cite sources, require citations to be from provided context, add a verification step (LLM checks each claim against chunks), use self-RAG to evaluate own answer, fall back to “I don’t know” if confidence is low.

RAG Architecture Patterns

RAG Architecture Patterns

The four evolutionary stages

Naive RAG (the baseline)

Advanced RAG — pre and post retrieval optimization

Pre-retrieval techniques

Step-back prompting and query routing

Post-retrieval techniques

Modular RAG

Agentic RAG

Trade-offs

Specialized patterns

Self-RAG

Corrective RAG (CRAG)

Multi-vector retrieval

Parent-document retrieval

Graph-augmented RAG

Production architecture (typical stack)

Failure modes

Cost vs quality knobs

Common interview confusions

Related

Interview angle 8