Design multi-tenant vector search

The brief: document search over 10M chunks belonging to several hundred customers. A tenant must never see another tenant’s content. p95 under a second, and citations back to the source document.

The interesting part is not the vector search. It is that isolation and approximate nearest neighbour search pull against each other: ANN indexes are built to return approximately the closest vectors in the whole index, and a tenant filter is a request to ignore most of it.

Clarify first

1. How many tenants, and how skewed? Fifty tenants with a roughly even split is a different design from ten thousand where one is 60% of the corpus. Skew decides whether per-tenant indexes are viable.
2. Is isolation a compliance boundary or a product rule? If a regulator audits it, the answer must be physical separation or row-level security you can demonstrate, not an application WHERE clause.
3. Do tenants ever share documents? Shared global content plus private content is a two-corpus problem, and merging results correctly is its own ranking question.
4. Write pattern. Continuous ingestion changes the index choice: IVFFlat clusters at build time and degrades as data drifts away from the centroids.

The second question is the one that separates the answers. “Isolation” that means a bug could expose data and isolation that means the architecture cannot express the query are different systems and different costs.

The three shapes

 A. shared index + filter    B. index per tenant     C. shard per tenant
    one table                   one table               one database
    WHERE tenant_id = ?         partial indexes         separate connection
    cheapest, weakest           middle                  strongest, priciest

For several hundred tenants on Postgres, the working answer is usually B: one table partitioned by tenant_id, each partition carrying its own HNSW index, so the filter becomes partition pruning rather than post-filtering.

CREATE TABLE chunks (
    id          bigserial,
    tenant_id   int    NOT NULL,
    document_id bigint NOT NULL,
    content     text   NOT NULL,
    embedding   vector(1536) NOT NULL
) PARTITION BY LIST (tenant_id);

CREATE TABLE chunks_t42 PARTITION OF chunks FOR VALUES IN (42);
CREATE INDEX ON chunks_t42 USING hnsw (embedding vector_cosine_ops);

The query planner prunes to one partition, and the ANN search then runs over that tenant’s vectors only — which is the property you want, because it makes recall independent of how small the tenant is.

Why the naive version fails

A single shared index with WHERE tenant_id = 42 post-filters: the index returns roughly ef_search candidates from the whole corpus, and the filter then discards the ones belonging to other tenants. A tenant holding 0.5% of the corpus gets back a handful of rows for a LIMIT 10, and the smaller the customer the worse their search is. Detail and mitigations in pgvector in production.

Naming this failure unprompted is most of the signal in this question.

Where the leak actually happens

Isolation fails in the places that are not the query. Enumerate these; it is what distinguishes a security answer from a filtering answer.

• The embedding cache. Keyed by text hash alone, a cache returns one tenant’s vector for another’s identical query. Harmless for vectors, not for a cached result set. Key by (tenant_id, hash).
• The reranker. A cross-encoder called with candidates from a leaky retrieval step launders the leak into a ranked answer.
• Logs and traces. Chunk text in a span attribute crosses into a shared observability backend that support staff read. Redact or hash.
• The eval set. Golden questions built from real customer documents, stored in a shared repo, is a data export nobody reviewed.
• The LLM prompt. Never rely on an instruction that says “only use documents for tenant 42”. It is not an access control.

Enforcing it so it cannot be forgotten

Put the tenant on the connection, not in every call site:

async def tenant_session(tenant_id: int) -> AsyncIterator[AsyncSession]:
    async with async_session() as s:
        await s.execute(
            text("SET LOCAL app.tenant_id = :t"), {"t": tenant_id}
        )
        yield s

With row-level security keyed on app.tenant_id, a forgotten WHERE clause returns nothing rather than everything — the failure becomes loud. The application filter stays as well; defence in depth, not either/or. The Postgres-side mechanics are in Multi-tenant data isolation.

Then test it. One test per boundary, asserting that tenant A’s query returns zero rows from tenant B’s fixtures, is worth more than any amount of review — the failure is silent, and the blast radius is every customer at once.

The arithmetic

10M chunks at 1536 dimensions, float32: 10e6 * 1536 * 4 ≈ 61 GB of raw vectors, plus HNSW graph overhead of roughly 30-50%. That does not fit in the RAM of a modest instance, which drives three decisions:

• halfvec halves storage for a small recall cost.
• Binary quantisation with rescoring cuts it much further, at more.
• Partitioning helps memory too — only the partitions being queried need their index pages resident, and most tenants are idle at any moment.

Per-tenant indexes have a floor: HNSW on a 200-row partition is pointless overhead, so route small tenants to exact search. Under a few thousand vectors, a sequential scan with a distance sort is faster than an index, and exact.

Failure modes to name

• The noisy tenant. One customer’s bulk re-ingest starves everyone. Queue ingestion per tenant with a concurrency cap.
• Partition count. Postgres handles thousands of partitions but planning time grows; past a few thousand tenants, group small ones into a shared partition with a filter and accept the cliff for them.
• Skewed recall. Report recall per tenant, not globally. A global 95% can hide a 60% for your smallest twenty customers.
• Deleting a tenant. With partitions it is a DROP TABLE, which is also the answer to a deletion request you have to evidence.

Related

• pgvector in production
  • the filtered-query cliff and index parameters.
• Hybrid search and reranking
  • tenant isolation as a retrieval concern, and reranking.
• Multi-tenant data isolation
  • the three isolation models and row-level security.
• Design: a RAG assistant over company documents - the retrieval pipeline this sits inside.
• Sharding and Partitioning
  • partitioning mechanics and pruning.