AI & ML / RAG & embeddings / 11_embedding_fine_tuning.md

Fine-tuning embeddings

Updated 5 interview angles 4 min read source
On this page8
  1. What you are actually training
  2. Where the pairs come from
  3. The training loop
  4. Measure retrieval, not loss
  5. The operational cost
  6. The alternatives, ranked by effort
  7. Related
  8. Interview angle

Fine-tuning embeddings

The retrieval lever people skip. When off-the-shelf embeddings underperform on your corpus, the usual response is a better reranker or a bigger model — fine-tuning the embedder is often cheaper and helps more.

It is worth reaching for when your domain has vocabulary the general model never learned: internal product names, part numbers, legal or clinical terms, or a jargon where two phrases mean the same thing to your users and nothing to a general model.

What you are actually training

Contrastive learning: pull matching pairs together in the vector space, push non-matching ones apart.

text 
query ────┐
          ├── close      (positive)
doc A ────┘

query ────┐
          ├── far        (negative)
doc B ────┘

The data you need is (query, relevant passage) pairs. Not labels, not categories — pairs.

Where the pairs come from

This is the whole project; the training is the easy part.

Source Quality
Search logs query → clicked doc; best, free
Support tickets question → resolving article
Generated questions one per chunk; synthetic
Manual annotation expensive, small

Search logs are the gold mine if you have them. A click after a query is a weak relevance label, and weak labels in volume beat a few perfect ones.

If you have none, generate: ask a model to write the questions each chunk answers, then embed the question and pair it with its source chunk. It works, with the usual synthetic-data caveats — see Distillation and synthetic data.

Hard negatives are what make it work

A random negative is easy: the model already knows an unrelated document is unrelated, so it learns nothing. Hard negatives — documents your current retriever ranks highly but which are wrong — are where the signal is.

python 
# Mine hard negatives from the current index
candidates = index.search(query, k=50)
negatives = [
    c for c in candidates if c.id not in positives
][:5]

Gotcha: mined negatives contain false negatives — genuinely relevant documents that simply were not labelled. Training on those teaches the model that a correct answer is wrong. Skip the very top ranks when mining, or use a judge to filter.

The training loop

python 
from sentence_transformers import (
    SentenceTransformer, losses, InputExample,
)

model = SentenceTransformer("base-model")
examples = [
    InputExample(texts=[q, pos, neg])
    for q, pos, neg in triples
]
loss = losses.MultipleNegativesRankingLoss(model)

MultipleNegativesRankingLoss is the workhorse: it treats the other passages in the batch as negatives, so you get many negatives for free and only need positive pairs. Larger batches give more in-batch negatives and better results, which makes batch size the parameter that matters most.

A few thousand pairs and an hour on one GPU is a realistic scale. This is not a large training job.

Measure retrieval, not loss

The loss going down means very little. Evaluate the thing you care about: recall@k and MRR on a held-out set of queries, against the base model as the baseline.

Expect the win to be domain-shaped: large on jargon-heavy corpora, small on general prose where the base model was already good. If you see a 2-point gain, a reranker was probably the better spend.

The operational cost

Everything from What are embeddings? about changing models applies, because you have changed the model:

  • Re-embed and re-index the entire corpus. Vectors from the old and new embedder are not comparable.
  • Version the index and cut over, rather than mixing.
  • You now own a model. It needs storage, serving, and retraining when the domain drifts — which is a real commitment against a hosted API.

That last point is the honest counterargument, and worth raising yourself.

The alternatives, ranked by effort

  1. Better chunking. Usually the biggest retrieval win, and free.
  2. Hybrid search plus a reranker. A cross-encoder captures much of what fine-tuning would, without a new model to own.
  3. A different off-the-shelf embedder, evaluated on your data.
  4. Fine-tune, once the above are exhausted and the domain is genuinely unusual.

Naming that order is the senior answer: fine-tuning the embedder is a real tool and it is fourth, not first.

Interview angle 5

  • “Retrieval is poor on our corpus. What would you try?” - chunking first, then hybrid search with a reranker, then a different off-the-shelf embedder, and only then fine-tune the embedder. Naming that order matters: fine-tuning is a real tool and it is fourth.
  • “What does fine-tuning an embedder need?” - (query, relevant passage) pairs, not labels. Search logs are the best source because a click is a weak relevance signal available in volume; support tickets work too; generating a question per chunk is the fallback.
  • “What are hard negatives and why do they matter?” - documents the current retriever ranks highly but which are wrong. A random negative teaches nothing because the model already separates unrelated text. The catch is false negatives — genuinely relevant documents that were never labelled — so skip the very top ranks when mining.
  • “How do you know it worked?” - recall@k and MRR on held-out queries against the base model, not the training loss. Expect a large win on jargon-heavy corpora and a small one on general prose, where a reranker would have been the better spend.
  • “What does it cost operationally?” - a full re-embed and re-index, since old and new vectors are not comparable, plus you now own a model that needs serving and retraining as the domain drifts. That is the real argument against it.
Previous
pgvector in production