RAG on large data volumes

Article.

The pipeline does not change at scale: parse, chunk, embed, index, retrieve, rerank, generate. What changes is that four things stop being free. Vectors stop fitting in memory, re-embedding becomes a project, a query fans out to many shards, and, the one most people miss, retrieval quality drops as the corpus grows even with a perfect index.

A senior answer starts with the arithmetic, because “large” means different things at 2 million chunks and at 2 billion.

Size it before designing it

Count chunks, not documents

The unit that costs memory and money is the chunk. Ten million documents at twenty chunks each is 200 million vectors:

200M chunks x 1024 dims
  float32  4 bytes/dim    ~820 GB
  int8     1 byte/dim     ~205 GB
  binary   1 bit/dim       ~26 GB
  binary at 256 dims        ~6 GB

embedding, 400 tokens/chunk  80B tokens
  at $0.02 per 1M tokens    ~$1,600

Two conclusions fall out. Memory decides the architecture: 820 GB of float32 means a cluster, 26 GB of binary vectors fits one machine. And embedding is cheap in dollars and expensive in time: $1,600 at the 2026 list price of a small hosted model (the provider’s batch tier halves it), but 80 billion tokens behind a rate limit of a few million tokens a minute is days to weeks of wall-clock.

The other numbers to ask for

• Query rate and latency target. 5 queries a second with a 2 s budget and 500 a second at 200 ms are different systems on the same corpus.
• Churn. A static archive is built once; a corpus where 5% changes daily needs incremental ingestion, deletes and tombstone cleanup (Ingestion pipelines and keeping an index in sync).
• Filters. Tenant, source, date, permissions. They decide how to shard, and, as the next section shows, they are a quality tool.

Quality drops as the corpus grows

This is the mechanism to know, because it is invisible in a pilot. Retrieval that worked on 100,000 chunks gets worse on 10 million, and no index setting fixes it.

Distractors, measured

A run on a synthetic corpus: 200 questions with known answer chunks, the same questions at every size, and exact search, so no ANN error is involved. Chunks cluster by subtopic, and a question says less than its answer does:

answers, t, detail, src = corpus(Q)
queries = unit(subs[t] + 0.06 * detail + 0.3 * noise(Q))   # a question says less
docs = answers                                               # than its answer does
for n in [10_000, 100_000, 1_000_000]:
    docs = np.vstack([docs, corpus(n - len(docs))[0]])
    rank = rank_of_answer()          # how many chunks outscore the right one

exact search; where does the right chunk rank?
     10,000 chunks: top-1 0.97   top-10 1.00
    100,000 chunks: top-1 0.79   top-10 0.99
  1,000,000 chunks: top-1 0.57   top-10 0.90
  1M filtered to one source:
                    top-1 0.98   top-10 1.00

The right chunk’s lead over its near neighbours does not change with corpus size. The number of near neighbours does: every subtopic gains members as the corpus grows, so more chunks land close enough to outscore the answer by chance. Old versions of the same page, the same policy for five regions, and templated documents are where real corpora make these.

Shrink the search space per query

The last row is the fix. Restricting the same 1M corpus to the question’s source (one of 100) brought it back to small-corpus quality. At scale, a metadata filter is a relevance tool, not just a speed tool:

• Filter or route by what the request already knows: tenant, product, region, date range, document type.
• Retrieve hierarchically: find the right documents first (by title, summary or a document-level vector), then the chunks inside them.
• Deduplicate at ingest: keep the current version of a page, collapse near-duplicates, drop superseded documents. Index fewer, better chunks.
• Retrieve deeper and rerank: a cross-encoder over 100-200 candidates separates what a bi-encoder’s similarity cannot (Hybrid search and reranking).

The consequence for evaluation: a golden set must run against the full corpus. Recall measured on a 100k pilot overstates production recall, and the gap grows with the corpus.

Memory: quantize, keep originals on disk, rescore

What each step buys

The pattern every large deployment converges on: search compressed vectors held in RAM, then rescore a shortlist with full-precision vectors read from disk. Hugging Face measured it on MTEB retrieval (NDCG@10, 2024):

The model matters more than the method: with the same rescoring, binary kept 96.5% for mxbai-embed-large but 74.8% for e5-base-v2. Binary works on models trained or known to tolerate it, and is a guess otherwise.

Measure the right recall

The same 1M synthetic corpus, quantized, scored two ways: overlap with the exact float32 top-10 (what ANN benchmarks report), and whether the answer is in the top 10 (what users feel):

Int8 lost almost nothing users would notice. Binary on these synthetic vectors is far worse than the published numbers for good models, which is the point of the table: measure on your embeddings, at your corpus size, with the task metric. Rescoring depth is the dial, as ef_search is for HNSW.

Fewer dimensions, and disk

• Matryoshka embeddings are trained so a prefix of the vector is itself a usable embedding. OpenAI’s text-embedding-3-large cut to 256 dimensions scored 62.0 on MTEB against 61.0 for ada-002 at 1536 (OpenAI’s January 2024 figures). Truncate, then renormalise.
• Stack them: 1024 dimensions cut to 256, then binary, is 32 bytes a vector. Rescore with the full vector from disk.
• DiskANN keeps the graph and full vectors on SSD and compressed vectors in RAM. The paper served a billion points from one 64 GB, 16-core machine at over 5,000 queries a second, under 3 ms mean latency, 95%+ recall@1 (SIFT1B, NeurIPS 2019). Milvus and pgvectorscale ship variants.

Mechanics of HNSW memory and product quantisation are in ANN indexes and how HNSW works; Qdrant’s configuration in Qdrant in production.

Sharding and fan-out

Split by key or split at random

• By key (tenant, source, region): a filtered query touches one shard. Cheap and fast, and it gives you the relevance win above. The risk is a hot shard: one tenant with half the data.
• At random (hash of the chunk id): data spreads evenly, and every unfiltered query must ask every shard, then merge.

Most systems do both: key-based partitions for tenants and sources, hashed shards inside a partition that outgrows one node.

The slowest shard sets the latency

A scatter-gather query waits for its slowest shard. Simulated with each shard’s latency drawn from the same distribution (p50 8 ms):

 1 shards: p50   8.0 ms   p99  33.0 ms
 4 shards: p50  14.7 ms   p99  43.9 ms
16 shards: p50  22.4 ms   p99  55.1 ms
64 shards: p50  31.9 ms   p99  69.6 ms

At 64 shards the median query is as slow as a single shard’s p99, because it almost always contains one slow shard. More shards add capacity and cost tail latency; replicas and hedged requests buy it back.

Each shard returns k, not k divided by shards

The global top-10 could all live on one shard, so every shard returns its own top 10 and the coordinator merges. Bound the wait, and say when the answer is partial:

async def search_all(shards, vector, k: int, deadline_s: float = 0.15):
    tasks = [asyncio.create_task(s.search(vector, k)) for s in shards]
    done, pending = await asyncio.wait(tasks, timeout=deadline_s)
    for t in pending:
        t.cancel()                          # a slow shard does not hold the query
    hits = [h for t in done if not t.exception() for h in t.result()]
    partial = len(done) < len(shards) or any(t.exception() for t in done)
    return heapq.nlargest(k, hits, key=lambda h: h.score), partial

Log partial. A rising rate of partial answers is a capacity alarm that users experience as “the assistant got worse”.

Ingestion and re-embedding at volume

Bulk load, then build

Inserting 200 million vectors one at a time into a live HNSW index is the slowest way to build it. Bulk load first and build the index after (the pgvector README says exactly this); on segment-based stores such as Milvus, size segments so sealing and index builds keep pace with ingest (Milvus in production).

async def embed_corpus(batches, embed, store, checkpoint, workers=8):
    done = await checkpoint.load()            # batches finished by an earlier run
    todo = iter([(i, b) for i, b in enumerate(batches) if i not in done])
    async def worker():
        for i, part in todo:                  # one shared iterator: no task per batch
            vectors = await embed([c.text for c in part])
            await store.upsert_many(          # deterministic ids: safe to retry
                (c.chunk_id, v, c.meta) for c, v in zip(part, vectors))
            await checkpoint.mark_done(i)     # batches finish out of order
    async with asyncio.TaskGroup() as tg:     # one failure stops the other workers
        for _ in range(workers):
            tg.create_task(worker())

workers is the concurrency bound, sized to the provider’s rate limit. The checkpoint records finished batches, not “the last id”, because concurrent batches finish out of order; a crashed run restarts with only the missing ones, and deterministic chunk ids make a repeated batch harmless.

Re-embedding is a migration with double storage

A new embedding model means a shadow index built alongside the live one (Ingestion pipelines and keeping an index in sync). At scale, plan for two full indexes during the cutover, a backfill measured in days, and dual writes so the shadow is current when you switch. Pin the embedding model version: a provider silently updating a model is a re-embed you did not schedule.

Where it breaks in production

• Recall measured on the pilot. The golden set passed on 100k chunks; at 20M, near-duplicates crowd the answers out. Evaluate on the full corpus.
• The hot shard. One customer owns 40% of the chunks, and their shard sets everyone’s p99. Split large tenants; keep small ones together.
• Permission filters that do not scale. A user in 5,000 groups produces a filter the index cannot apply efficiently. Precompute access tokens per chunk, or partition by the coarsest permission boundary (Design multi-tenant vector search).
• Tombstones from churn. Deletes accumulate in graph indexes and degrade latency and memory until a rebuild or compaction runs.
• The reranker becomes the bill. 200 candidates through a cross-encoder at 300 queries a second is 60,000 pairs a second. Rerank fewer, or cache.
• Silent partial answers. A shard times out, the merge returns nine shards’ results, and nobody knows. Return and log a partial flag.

The trade not taken

The tempting design is one giant index: embed everything, put it in a distributed vector database, let ANN sort it out. It scales the storage and leaves the quality problem untouched, because the distractors are still in the candidate set.

The design that holds up is route, then search: partition by what queries already know, deduplicate at ingest, retrieve documents before chunks, and size each search space so it behaves like a small corpus. A single node with quantized vectors and an on-disk index handles hundreds of millions of vectors; reach for a cluster when the arithmetic says so, not before.

Related

• ANN indexes and how HNSW works - HNSW memory, filtering, tombstones
• Hybrid search and reranking - fusion and rerank depth
• Ingestion pipelines and keeping an index in sync - change detection, re-embed migrations
• Milvus in production - segments, DiskANN, partitions
• Agentic RAG and evaluating retrieval - recall@k and golden sets
• Elasticsearch — Vector Search and kNN - kNN in Elasticsearch
• The LLM gateway and caching - caching above retrieval