Vector Databases

Article.

A vector database stores embeddings (typically 384-3,072 dimensions) with their metadata and answers “which stored vectors are nearest to this one?” in milliseconds. It is the storage layer under RAG, semantic search and recommendations. The interview question behind it is rarely “name five vector databases”. It is what does a vector database add over an index library, and when do you actually need one?

Embeddings themselves are in What are embeddings; how the index works is ANN indexes and how HNSW works.

What it does

# Index a chunk: the vector, plus the payload you will filter on
vector_db.upsert(
    id="doc_42#3",
    vector=model.encode("Refunds are issued within 14 days."),
    payload={"tenant_id": "acme", "source": "policy.pdf", "year": 2026},
)

# Query: the nearest vectors that also pass the filter
hits = vector_db.search(
    vector=model.encode("how long do refunds take?"),
    filter={"tenant_id": "acme"},
    limit=5,
)

Exact nearest-neighbour search is a scan of every vector, fine below roughly a hundred thousand and too slow above a few million. So databases use an approximate index, almost always HNSW, and expose recall (the share of the true top-k you get back) as a dial traded against latency.

The mechanism: a library plus what a database adds

The index itself is a solved, free problem: FAISS, hnswlib and the HNSW code inside every database build the same kind of graph. What you pay a database for is everything around it:

Three mechanisms explain most of the behaviour you will meet:

1. Deletes are tombstones. A graph node cannot simply be removed, because its neighbours route through it. The node is marked dead and skipped, and a background compaction or rebuild reclaims it. Heavy churn means a slower, larger index until that runs.
2. Writes are not instantly searchable in segment-based engines (Milvus, Elasticsearch): new vectors sit in a growing segment until it is sealed and indexed (Milvus in production).
3. Filtering has to cooperate with the graph. A greedy walk through neighbours stalls when most neighbours fail the filter; databases that apply the filter during traversal (Qdrant’s filterable HNSW, pgvector 0.8’s iterative scans) are solving exactly that.

The metric, and a myth about it

“Use the wrong distance metric and quality silently tanks” is half true. For normalized vectors, cosine, dot product and L2 produce the same ranking, because with unit length they are monotonic functions of each other. Measured on 100,000 random 384-dimensional vectors:

normalized vectors (unit length):  top-10 shared with cosine: dot 100%, L2 100%
raw vectors, lengths vary 0.5x-2x: top-10 shared with cosine: dot 20%,  L2 0%

Most text embedding models return normalized vectors, which is why the wrong metric usually costs nothing there. It matters when vectors are not normalized (some image and recommendation embeddings, or vectors you averaged yourself): dot product then favours long vectors, and L2 ranks something else again.

Normalize once at index time and the choice stops mattering; in pgvector the operator must still match the index opclass, or the index is silently unused (pgvector in production).

Filtering

Almost every production query is “similar and belongs to this tenant, this year, this document type”. The three strategies behave very differently for a selective filter. One tenant out of 50 (2% of 100,000 vectors), wanting 10 results:

search top-100, then filter:   2.2 of 10 results on average
search top-1000, then filter:  10.0 of 10 results on average
brute force over the tenant's own 2,003 vectors: 0.14 ms per query

• Post-filtering silently returns too few results unless you over-fetch by roughly the inverse of the filter’s selectivity.
• Filter inside the index (filterable HNSW, iterative scans) keeps the walk going through matching nodes; it is what to look for in a database.
• Brute force on the subset is exact and, for a selective filter, faster than any graph: 2,000 vectors is a fraction of a millisecond.

# Qdrant: the filter participates in the HNSW traversal
hits = client.query_points(
    collection_name="docs",
    query=query_vector,
    query_filter=models.Filter(must=[
        models.FieldCondition(key="tenant_id", match=models.MatchValue(value=tenant)),
        models.FieldCondition(key="year", range=models.Range(gte=2025)),
    ]),
    limit=10,
)

Multi-tenancy

The shared-plus-filter design is the common default and the riskiest: every query must carry the tenant filter, so put it in one repository function that cannot be called without a tenant id. The design in depth is Design multi-tenant vector search.

The landscape

pgvector: the default to beat

CREATE EXTENSION vector;

CREATE TABLE chunks (
    id         bigserial PRIMARY KEY,
    tenant_id  bigint NOT NULL,
    content    text,
    embedding  vector(1536)
);
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

SELECT id, content
FROM chunks
WHERE tenant_id = 42
ORDER BY embedding <=> $1
LIMIT 10;

Vectors, joins, transactions and backups in the database you already run, and a delete that is a DELETE. It stays comfortable into the low tens of millions of vectors per node; past that, memory and index build time push you towards a dedicated engine (RAG on large data volumes).

Where it breaks in production

• Stale and orphaned vectors. A document is edited or deleted and its chunks stay searchable, because ingestion inserted new ids instead of replacing old ones. Deterministic chunk ids and a reconciliation job fix it (Ingestion pipelines and keeping an index in sync).
• Re-embedding is a migration. Vectors from two models are not comparable and there is no conversion; changing models means re-embedding everything into a second index and cutting over.
• A filter that starves results. Post-filtering on a selective tenant returns 2 results instead of 10, with no error, and the answer quality drops only for small customers.
• Tombstone build-up. High-churn collections slow down and grow until compaction runs; schedule it, and watch the deleted-to-live ratio.
• The tenant filter forgotten once. One code path without it is a data leak across customers, found by a customer.
• Recall never measured. Index parameters are tuned for latency and nobody compares against exact search on a held-out set, so recall drifts as the collection grows.

The trade not taken

The tempting first move is a dedicated vector database for a corpus of fifty thousand chunks. At that size an exact numpy scan is a few milliseconds, and pgvector puts the vectors in the transaction that wrote the document, with the backups, access control and tenant column you already have. A separate engine adds a second datastore to keep in sync, which is where the stale-vector and leaked-tenant bugs above come from.

Reach for a dedicated engine when the numbers say so: vectors past what one Postgres node holds comfortably, filtering that pgvector’s iterative scans cannot keep fast, or a team that wants hybrid search and quantization as managed features rather than SQL.

Related

• ANN indexes and how HNSW works - how the index works, and why filters fight it
• pgvector in production - operators, opclasses, the filtered-query cliff
• Qdrant in production - filterable HNSW, named vectors, quantization
• Milvus in production - segments and consistency levels
• Hybrid search and reranking - BM25 plus vectors, fused
• RAG on large data volumes - memory arithmetic, sharding, quantization at volume