Hybrid search and reranking

Article.

The two upgrades that take a RAG demo to production. As of 2026 the consensus is settled: hybrid retrieval is the minimum viable baseline, and hybrid retrieval followed by neural reranking beats any single-stage method by a wide margin.

Both are easy to add and easy to add badly, and the ways they fail in production are about latency and filtering rather than relevance.

Why pure vector search is not enough

Dense embeddings capture meaning and lose precision on exact tokens.

BM25 nails exact-match queries — product codes, error codes, proper nouns, rare technical terms — and cannot handle paraphrase. Dense retrieval handles concepts and paraphrase and systematically underweights a rare exact token, because that token is one small contribution to a single averaged vector.

Neither is sufficient alone, and their failure cases are complementary. That is the whole argument, and it is why “we’ll just use embeddings” is the most common reason a RAG system works in the demo and not on real queries.

Fusing the two

BM25 scores and cosine similarities are on incomparable scales. Normalising them is possible and fragile: the score distributions shift with corpus and query, so a weighting tuned in March is wrong in June.

Reciprocal Rank Fusion sidesteps calibration entirely by discarding the scores and using only ranks:

def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    """Fuse ranked lists by reciprocal rank. Scale-independent."""
    scores: dict[str, float] = collections.defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores, key=scores.__getitem__, reverse=True)

fused = rrf([bm25_search(query, top=50), vector_search(query, top=50)])

k=60 is the conventional constant. Its job is to flatten the contribution of the very top ranks — with k=0, rank 1 scores 1.0 and rank 2 scores 0.5, so a single retriever’s top hit dominates everything. At k=60 those become 0.0164 and 0.0161, so agreement across retrievers matters more than any one system’s confidence.

Start with RRF. Weighted score fusion can beat it once tuned, and needs retuning whenever either retriever, the embedding model, or the corpus changes.

The Postgres detail worth getting right

Teams already on Postgres often combine pgvector with full-text search in one query, and that is frequently the right production answer — one system, one transaction, no extra infrastructure.

Be precise about what it gives you though: Postgres core full-text search is not BM25. ts_rank and ts_rank_cd are term-frequency-based ranking functions without BM25’s document-length normalisation and saturation. Good enough to be a strong lexical arm in a hybrid setup, but if a benchmark or an interviewer says BM25 specifically, core Postgres is not it — that needs an extension such as ParadeDB’s pg_search.

Reranking

Retrieval optimises recall over a huge corpus, cheaply. Reranking optimises precision over a small candidate set, expensively.

candidates = hybrid_search(query, top_k=50)     # cheap, recall-oriented
reranked = cross_encoder.rank(query, candidates) # expensive, precision-oriented
context = reranked[:5]                           # what the model actually sees

Bi-encoder versus cross-encoder

The mechanism behind the accuracy gap: a bi-encoder must compress each document into a single vector before it has seen the query. Whatever the query turns out to ask about, that decision was already made.

A cross-encoder processes query and document in one forward pass, so attention can relate a specific query term to a specific span. That recovered information is what reranking buys, and it is why a better embedding model never fully substitutes for it.

The cost is exactly why it is a second stage. You cannot cross-encode a million documents per query; fifty is trivial.

Reranking is usually the single highest-return change to a mediocre RAG system — ahead of better chunking, ahead of a better embedding model.

Late interaction, the third option

ColBERT sits between the two. Instead of one vector per document, it stores one vector per token, and scores a pair with MaxSim: for each query token, take the best-matching document token, then sum.

bi-encoder      one vector per doc     -> cheap, coarse
late interaction  one vector per token -> mid, keeps term detail
cross-encoder   joint forward pass     -> expensive, most accurate

Because the document vectors are still precomputed, this is a retrieval method, not a rerank-only one — the query-document interaction is deferred to scoring rather than skipped, hence “late interaction”. It recovers much of the term-level precision a bi-encoder discards, which is why it does well on exactly the queries dense retrieval fumbles: rare terms, codes, names.

The cost is storage. One vector per token is roughly two orders of magnitude more than one per chunk, before compression. PLAID and the ColBERTv2 residual compression bring it down, but it remains the reason most teams reach for hybrid plus a cross-encoder instead: similar effect, no new index to operate.

Worth naming in an interview as the option you considered and rejected, with the storage reason. Reach for it when lexical precision matters and you cannot afford a cross-encoder’s per-candidate latency.

Choosing candidate count

Retrieve 30-100 for reranking, keep 3-10 for context.

Tune the first number by measuring recall@k: if the correct chunk is rarely in the top 50, reranking cannot save you, and your problem is retrieval. This ordering matters — adding a reranker before you know your recall is guessing at which stage is broken.

Keeping too many after reranking actively hurts. Extra chunks dilute attention and push relevant content into the weak middle of the context window; see Context windows.

MMR, when the top-k is redundant

Relevance ranking alone returns near-duplicates: five chunks from the same section of the same document, all correct, all saying the same thing. The answer then rests on one source and misses the other four the question needed.

Maximal Marginal Relevance re-scores candidates against what has already been selected, so each pick trades relevance against novelty:

def mmr(query_vec, cands, k=5, lam=0.7):
    """lam=1.0 is pure relevance, lam=0.0 is pure diversity."""
    chosen = []
    while cands and len(chosen) < k:
        best = max(cands, key=lambda c: (
            lam * cos(query_vec, c.vec)
            - (1 - lam) * max((cos(c.vec, s.vec) for s in chosen),
                              default=0.0)
        ))
        chosen.append(best)
        cands.remove(best)
    return chosen

Use it when one query legitimately has several distinct answers — comparisons, “what are the options”, multi-aspect questions — and skip it when the corpus is already diverse, where it only demotes the best chunk. lam around 0.7 is a sane default; below 0.5 it starts returning off-topic content for the sake of being different.

Deduplicating by document ID first is cheaper and fixes the common case. Reach for MMR when near-duplicates survive that, which is what happens with overlapping chunks of the same source.

Where this breaks in production

The latency budget

This is the constraint the architecture is really shaped by, and it rarely appears in tutorials.

rewrite query      ~200ms   (an LLM call)
BM25 + vector       ~30ms   (parallel)
RRF fuse             ~1ms
cross-encode 50   ~150ms    (GPU, batched)
                  ~2000ms+  (CPU, unbatched)
generate           ~2000ms

Two consequences follow. Batch the cross-encoder — fifty pairs in one batched forward pass, not fifty sequential calls; the difference is an order of magnitude. And run BM25 and vector search concurrently, since they are independent and the slower one sets the cost.

If reranking on CPU is your only option, a smaller reranker over 20 candidates beats a large one over 50, and both beat skipping the stage.

The reranker is a dependency that can fail

A hosted reranker is a network call in your critical path. When it times out, the wrong behaviour is to fail the request:

try:
    ranked = await rerank(query, candidates, timeout=0.5)
except (TimeoutError, RerankerError):
    log.warning("reranker unavailable, serving fused order")
    ranked = candidates          # RRF order is a decent fallback

Degrade to the fused ranking. It is worse and it is not broken, which is the right trade for a relevance improvement — a reranker outage should cost quality, never availability.

Filtering fights the ANN index

The note that filtering is a hard constraint is only half of it. Where the filter runs relative to the graph traversal changes what you get back.

• Post-filter — retrieve top 50 by vector, then drop rows failing the filter. You may be left with three results, or none, because the filter was never part of the search.
• Pre-filter — restrict the candidate set first, then search. Correct counts, but an HNSW graph traversed under a restrictive filter can become effectively disconnected: the neighbours it wants to hop through are excluded, so recall silently drops.

Vector databases handle this differently, and it is a real selection criterion — Qdrant builds filterable HNSW so the filter participates in traversal rather than fighting it. In pgvector the planner may choose to ignore the index entirely under a selective filter and scan instead, which is correct but slow.

On a highly selective filter, a plain SQL scan often beats the vector index. If one tenant has 400 documents, filter to those 400 and brute-force the cosine distances; an ANN index earns its complexity at scale, not at 400 rows.

Tenant isolation is not a relevance signal

results = collection.query(
    query_embedding=emb,
    filter={"tenant_id": user.tenant_id, "status": "published"},
    top_k=50,
)

Retrieval leaking across tenants is a security incident, not a ranking bug. Enforce it in the query, never by trusting the model to ignore documents it should not have been shown — and test it, because the failure is silent and the blast radius is every customer. See Multi-tenant data isolation.

Recency as a ranking signal

For corpora where freshness matters - support tickets, news, changelogs, documentation - relevance alone ranks a correct 2019 answer above a correct 2026 one.

def time_weighted(candidates, half_life_days=30):
    now = utcnow()
    for c in candidates:
        age = (now - c.published_at).days
        c.score *= math.exp(-age / half_life_days)
    return sorted(candidates, key=lambda c: -c.score)

Apply it after fusion and before reranking, so it shapes the candidate set the cross-encoder sees. And keep the decay gentle: an aggressive half-life buries a definitive old document under a recent tangential one, which is the failure this is supposed to prevent running in reverse.

Contextual retrieval

A chunk stripped of its document loses meaning. “The policy applies for 30 days” — which policy?

Contextual chunking prepends a short generated description of where the chunk sits, before embedding:

Context: 2026 Refund Policy, digital goods section.
Chunk:   The policy applies for 30 days from purchase.

Both the embedding and the lexical index see the enriched text. It costs one cheap LLM call per chunk at ingest — one-time and offline, and prompt caching makes it inexpensive because the source document is a shared prefix across all of its chunks.

This addresses the most common RAG failure: retrieval returns a chunk that is topically right and contextually meaningless.

Query-side techniques

Query rewriting matters most in practice, because multi-turn chat produces queries that are meaningless standalone. Resolving “what about the second one?” against conversation history before retrieval is usually a bigger win than any index tuning — and it is the stage most systems skip.

The pipeline, assembled

query
  -> rewrite (resolve context, expand)
  -> hybrid: BM25 top 50 + vector top 50
  -> RRF fuse
  -> metadata filter (hard constraints)
  -> cross-encoder rerank
  -> top 3-10 into context
  -> generate with citations

Build it in that order and measure at each stage, because each one can only be evaluated against the one before it.

The trade not taken

Every stage above is latency and cost, and the alternative is to need fewer of them. If the corpus is small — a few thousand chunks — put the whole thing in the context window and skip retrieval altogether; long-context models make that viable more often than RAG advocates admit, and it removes an entire subsystem’s worth of failure modes.

And if the question is answerable from structured data, a SQL query beats retrieval outright: it is exact, cheap, and cannot hallucinate a row. RAG earns its complexity on large unstructured corpora, which is a narrower case than the enthusiasm suggests.

Related

• Chunking - what you are indexing in the first place
• Agentic RAG and evaluating retrieval - measuring recall@k and building a golden set
• Qdrant in production - filterable HNSW in practice