AI & ML / RAG & embeddings / 01_what_is_rag.md

What is RAG?

Updated 5 interview angles 5 min read source
On this page6
  1. The two pipelines
  2. Why not just fine-tune?
  3. Where RAG actually fails
  4. What to build first
  5. Where the depth is
  6. Interview angle

What is RAG?

Retrieval-Augmented Generation: fetch relevant text at query time and put it in the prompt, so the model answers from documents you control rather than from what it memorised during training.

The one-line version an interviewer wants: RAG turns a knowledge problem into a retrieval problem.

The two pipelines

RAG is two pipelines, not one. Confusing them is the most common interview mistake — indexing happens once per document, retrieval happens once per query.

text 
ingest (per document)    query (per request)
─────────────────────    ───────────────────
parse                    rewrite query
  |                        |
chunk                    retrieve top-k
  |                        |
embed                    rerank
  |                        |
index  ───────────────▶  assemble context
                           |
                         generate + cite

Everything expensive and slow is on the left. Everything latency-critical is on the right.

The loop, stripped of framework

python 
def answer(question: str) -> str:
    hits = index.search(embed(question), k=20)
    top = rerank(question, hits)[:5]
    context = "\n\n".join(d.text for d in top)
    return llm(PROMPT.format(ctx=context, q=question))

That is the whole idea. Retrieve wide, rerank narrow, generate from what survived. Everything else in this folder is a refinement of one of those four lines.

Gotcha: retrieve wide then narrow, never narrow then wide. k=5 straight from the vector index gives the reranker nothing to work with.

Why not just fine-tune?

RAG Fine-tuning
Teaches facts form and behaviour
Update reindex a document retrain
Cost storage and queries GPU hours
Citations yes no
Fails by retrieving wrong forgetting

The distinction that lands in an interview: fine-tuning changes how a model says things, RAG changes what it knows. A model that answers in the wrong format needs fine-tuning. A model that answers with the wrong facts needs RAG. They compose — most production systems use both.

See Fine-Tuning vs RAG vs Prompt Engineering.

Has long context replaced it?

No, and the reasoning matters more than the answer. Frontier models take hundreds of thousands of tokens, so “just paste the corpus” is now technically possible for small corpora.

Long context wins RAG wins
small fixed corpus unbounded corpus
every token is relevant you pay per token
one-off analysis per-request latency matters

Attribution is the deciding factor in regulated work: RAG knows which chunk produced a claim, a stuffed context window does not. In practice the two combine — retrieve first, then give the model generous context over what survived.

Where RAG actually fails

Retrieval, not generation. This is the highest-signal thing to say about RAG, because it redirects debugging away from prompt tweaking.

  1. The right chunk was never retrieved. Measure recall@k first. If the answer is not in the candidate set, no prompt fixes it.
  2. It was retrieved but ranked 40th. A reranker fixes this; a bigger k only hides it.
  3. It was split in half by chunking. The definition landed in one chunk and the qualifier in the next.
  4. It was retrieved and ignored. Long contexts lose the middle — put the strongest evidence first and last.
  5. Nothing relevant exists. The system must be able to say so; a RAG pipeline with no abstain path will invent an answer.

The debugging order follows the list: check recall before you touch the prompt.

The evaluation that follows from it

Because failures are retrieval failures, evaluate the two stages separately:

Stage Metric Answers
Retrieval recall@k was the evidence there?
Retrieval MRR / nDCG was it near the top?
Generation faithfulness is the answer grounded?
Generation relevance did it answer the question?

A system with 60% recall and perfect faithfulness is a retrieval project, not a prompting one. Details in Agentic RAG and evaluating retrieval.

What to build first

Build the naive version, measure it, and only then add machinery — every stage you add is latency and a new failure mode.

  1. Fixed-size chunks with overlap, one embedding model, top-k vector search.
  2. Measure recall@k on 30 real questions. This is the baseline.
  3. Add a reranker. It is almost always the largest single win.
  4. Add hybrid search if queries contain names, codes or identifiers that embeddings blur together.
  5. Add query rewriting if queries are conversational or underspecified.

Stages 3–5 are covered in Hybrid search and reranking and Chunking Strategies and Retrieval Techniques.

Note: as of 2026-08, LangChain and LangGraph are both 1.0+. Pre-1.0 tutorials import from langchain.llms and langchain.embeddings; those moved to provider packages such as langchain_openai. Quoting the old import paths in an interview dates you precisely.

Where the depth is

Interview angle 5

  • “What is RAG and why use it over fine-tuning?” - retrieve relevant context at query time and put it in the prompt. It gives current facts, citations and cheap updates; fine-tuning teaches form and behaviour and cannot be updated or cited. Knowledge is a retrieval problem, behaviour is a training problem.
  • “Walk through the pipeline.” - two pipelines: ingest (parse, chunk, embed, index) runs per document; query (rewrite, retrieve, rerank, assemble, generate with citations) runs per request. Naming rewrite and rerank as separate stages is what separates a production answer from a demo.
  • “Where do RAG systems usually fail?” - retrieval, not generation. Measure recall@k first: if the right chunk is not in the candidate set, no prompt engineering fixes the answer. The second most common failure is a chunk boundary splitting a fact from its qualifier.
  • “Has long context replaced RAG?” - no. RAG wins on cost, latency, unbounded corpora and attribution. Production systems usually retrieve first, then give the model generous context over what survived.
  • “How would you know your RAG system is good?” - evaluate retrieval and generation separately, because a faithful answer over the wrong documents still scores well end-to-end. Recall@k on a fixed question set is the number to move first.