AI & ML / Training & fine-tuning / 07_fine_tuning_vs_rag.md

Fine-tuning versus RAG

Updated 7 interview angles 7 min read source
On this page8
  1. What each one actually changes
  2. The mechanism: why fine-tuning does not teach facts
  3. The cost crossover
  4. The decision
  5. Where it breaks in production
  6. The trade not taken
  7. Related
  8. Interview angle

Fine-tuning versus RAG

Article.

“Should we fine-tune on our documentation?” is the most common wrong instinct in applied LLM work, and answering it well requires knowing what fine-tuning actually changes — which is not what most people assume.

The short version: RAG gives a model facts it can read; fine-tuning gives it habits it will follow. Almost every correct decision here follows from that distinction.

What each one actually changes

What it changes Updating
Prompt the context edit the text
RAG the context, at runtime reindex
Fine-tuning the weights retrain
Cost shape Good for
Prompt per token format, simple rules
RAG infra + per query facts, large corpora
Fine-tuning one-off + cheaper queries tone, task shape

Prompting and RAG are the same mechanism at different times: both put text in the window, one written by you and one fetched. Fine-tuning is categorically different — it moves the weights.

The mechanism: why fine-tuning does not teach facts

Training nudges weights to make the observed token sequences more likely. That process is very good at reinforcing patterns that appear consistently across many examples, and very bad at installing individual facts.

Consider fine-tuning on a support corpus containing, once:

text 
Enterprise plan customers get a 45-day refund window.

Against thousands of other examples, that single sequence produces a gradient update small enough to be noise. The model does not acquire a retrievable fact — it acquires a slightly stronger tendency to produce plausible refund-policy- shaped sentences.

Which is worse than not learning it, because the failure looks like this:

text 
Q: What is the refund window for Enterprise?
A: Enterprise gets a 30-day refund window.
   ^ confident, wrong

The tone is right, the format is right, the structure is right. Fine-tuning successfully taught everything except the fact, and it removed the very signal — hedging, uncertainty — that would have told a user not to trust it.

RAG has the opposite failure mode: if the document is not retrieved you get “I don’t know”, which is annoying and honest.

What it does teach well

Anything that is a pattern rather than a datum:

  • Output format and structure, reliably, without instructions in every prompt.
  • Tone and persona.
  • A narrow task shape — classification, extraction, a specific transformation.
  • Domain vocabulary and phrasing conventions.
  • Tool-calling conventions specific to your tools.

The test is: does this appear the same way in every training example? If yes, fine-tuning learns it. If it appears once, it does not.

The cost crossover

The decision is often framed as quality; in practice it is frequently economics, and the arithmetic is worth doing because it can go either way.

RAG adds retrieved context to every request. Fine-tuning pays once and then lets you send a shorter prompt, because the instructions and examples are in the weights.

text 
RAG         +2,500 tokens of context per request
fine-tuned  -1,500 tokens (no few-shot, shorter system)
            ---------------------------------------------
            4,000 input tokens per request, RAG spends

one-off     the training run, eval, and owning the pipeline

At a few thousand requests a month, the training run never pays back. At tens of millions, the per-request saving dwarfs it and fine-tuning is the cheaper system even ignoring quality.

Two things distort this and are usually forgotten: prompt caching makes the long RAG prompt much cheaper than the token count suggests, since the stable prefix is cached; and a fine-tune has to be re-run whenever the base model moves, which is a recurring cost the naive calculation omits.

The decision

In order, stopping at the first that works:

  1. Prompt. Free, instant to iterate, and solves more than people expect. If you have not tried few-shot examples, you have not tried.
  2. RAG. The moment the answer depends on facts that are private, large or changing. This is most business applications.
  3. Fine-tune. When prompting cannot get the behaviour reliably enough, when prompt length is a cost problem at scale, or when you need a small model to punch above its weight on one narrow task.

The third has a precondition people skip: you cannot tell whether a fine-tune helped without an eval set. Fine-tuning before you can measure is buying a change you cannot evaluate — see Building eval sets.

They compose, and that is usually the answer

The strongest real systems are both: fine-tune for the behaviour, retrieve for the facts.

text 
fine-tuned:  our house format, our terminology,
             our tool-calling conventions
retrieved:   what the refund policy says today

A model fine-tuned to answer in a rigid citation-carrying format, fed retrieved chunks, is measurably better than either alone — the fine-tune stops you spending tokens on format instructions, and retrieval keeps the facts current.

Where it breaks in production

Fine-tuning on documents. The failure at the top of this note. It is attractive because the data is right there and it feels like teaching. It produces a model that sounds like your documentation and invents its content.

Catastrophic forgetting. Train hard on a narrow task and general capability degrades — the model gets better at your classification and worse at everything, including instruction-following you were relying on. Mitigate by keeping the learning rate low, mixing in general examples, and using a parameter-efficient method that touches few weights; see LoRA and parameter-efficient fine-tuning.

Too little data. A few dozen examples will not move a model meaningfully, and people conclude fine-tuning does not work. The threshold for a narrow task is typically hundreds to low thousands of consistent examples, and consistency matters more than volume — contradictory examples teach the model that both behaviours are acceptable.

Version lock-in. Your fine-tune is bound to one base model. When that base is deprecated or a better one ships, you cannot carry the adapter across; you re-run training, re-evaluate and re-deploy. A RAG system upgrades its model with a config change. This is the most underrated cost, and it recurs.

No rollback story. A prompt change reverts in a deploy. A fine-tune reverts by keeping the previous model version live and repointing — which only works if you kept it and versioned the traffic split deliberately.

The trade not taken

The option nobody costs properly is doing neither: a stronger base model with a good prompt.

Model capability moves fast enough that a behaviour requiring a fine-tune last year is frequently promptable now. Before committing to a training pipeline — data curation, training runs, eval, versioning, re-running on every base upgrade — it is worth checking whether the current frontier model simply does the task.

The reverse is also true and less often said: for a narrow, high-volume, latency-sensitive task, a small fine-tuned model beats a large prompted one on cost and speed by a wide margin, and that is the strongest case for fine-tuning there is. Classification at a million requests a day is exactly this, and reaching for a frontier model there is the expensive mistake.

Interview angle 7

  • “Should we fine-tune on our documentation?” — almost certainly not. That is a retrieval problem. Fine-tuning teaches behaviour rather than facts, so you get a model that sounds like your docs and invents their content.
    • Follow-up: “Why does it not learn the facts?” — a fact appearing once in a corpus produces a gradient update indistinguishable from noise. What is reinforced is the pattern across all examples: the format and tone. The resulting error is confident and unhedged, which is worse than a retrieval miss saying “I don’t know”.
  • “So what is fine-tuning for?” — anything consistent across every example: output format, tone, a narrow task shape, domain phrasing, tool-calling conventions. The test is whether it appears the same way in every example.
  • “How do you decide between them?” — prompt first because it is free and solves more than expected; RAG when the answer depends on private, large or changing facts; fine-tune when prompting cannot get the behaviour reliably, or when prompt length has become a cost problem at scale.
    • Follow-up: “What has to exist before you fine-tune?” — an eval set. Without one you cannot tell whether the run helped, and you have bought a change you cannot measure.
  • “Is fine-tuning cheaper than RAG?” — it depends on volume. RAG spends tokens per request; a fine-tune spends once and shortens every prompt. At low volume the run never pays back; at very high volume it dominates. Adjust for prompt caching, which makes the long RAG prompt cheaper than it looks, and for having to re-run the fine-tune on every base-model upgrade.
  • “What is the most underrated cost of fine-tuning?” — version lock-in. The adapter belongs to one base model, so a better model shipping means retraining, re-evaluating and re-deploying. A RAG system changes model with a config edit.
  • “Your fine-tuned model got worse at everything else.” — catastrophic forgetting from training too hard on a narrow task. Lower the learning rate, mix in general examples, and prefer a parameter-efficient method that touches few weights.
  • “When is fine-tuning clearly right?” — a narrow, high-volume, latency-sensitive task where a small fine-tuned model beats a large prompted one on cost and speed. Classification at a million requests a day is the canonical case.
Previous
LoRA and parameter-efficient fine-tuning