AI & ML / RAG & embeddings / 13_milvus_in_production.md

Milvus in production

Updated 6 interview angles 5 min read source
On this page7
  1. The architecture that differentiates it
  2. Segments, and why search results lag writes
  3. Indexes
  4. Filtering
  5. Choosing it, honestly
  6. Related
  7. Interview angle

Milvus in production

Milvus is the vector database people reach for at the scale where Qdrant and pgvector stop being comfortable - hundreds of millions to billions of vectors, usually with GPU acceleration. Its architecture is the reason, and it is also the reason it is heavier to run.

The architecture that differentiates it

Milvus separates into four independent layers, each scaled on its own:

text 
access   -> proxy (stateless, load balanced)
coord    -> root / query / data / index
worker   -> query nodes, data nodes, index nodes
storage  -> object store + etcd + message queue

The consequence worth knowing: indexing and querying scale separately. A bulk re-index adds index nodes without touching query capacity, which is exactly the operation that degrades a single-process vector store.

Persistence is object storage (S3 or MinIO), metadata is in etcd, and the write path goes through a log broker - Pulsar or Kafka. That last piece is why Milvus is not a single binary: a production cluster has real dependencies.

Milvus Lite exists for local development and small embedded use, and Zilliz Cloud is the managed option. Choosing between Lite, standalone and distributed is the first deployment decision.

Segments, and why search results lag writes

Data lands in growing segments held in memory, which are sealed at a size threshold and then indexed. Growing segments are searched by brute force; sealed segments are searched via their index.

This produces the behaviour that confuses people first: a vector is searchable but slow immediately, and fast only after its segment seals and indexes.

Consistency is explicit, and the level chosen decides whether a just-inserted vector is visible:

Level Behaviour
Strong Waits for all writes; slowest
Bounded Staleness within a window; the default
Eventually Fastest, no guarantee

For a RAG pipeline that writes then immediately queries - an ingestion test, or a user uploading a document and asking about it - Bounded will intermittently miss the new data. That is a correctness bug that only appears under timing.

Indexes

Milvus exposes more index types than most alternatives, and picking one is a real decision rather than a default:

  • HNSW - the general answer. High recall, memory-resident.
  • IVF_FLAT / IVF_PQ - inverted file with optional product quantisation. Lower memory at a recall cost; nprobe trades speed against recall.
  • DiskANN - for datasets that exceed RAM, keeping most of the graph on SSD. This is the reason Milvus appears in billion-scale answers.
  • GPU indexes (CAGRA and others) - the GPU acceleration Milvus is known for, worth it when query volume is high enough to keep the card busy.
python 
collection.create_index(
    field_name="embedding",
    index_params={
        "index_type": "HNSW",
        "metric_type": "COSINE",
        "params": {"M": 16, "efConstruction": 200},
    },
)

The metric must match how the embeddings were trained. Cosine for normalised embeddings; using L2 on them silently degrades recall rather than erroring.

Filtering

Milvus supports scalar filtering alongside vector search, and as with any vector store the question is whether the filter applies before or during the search rather than after.

python 
res = collection.search(
    data=[qvec], anns_field="embedding",
    param={"metric_type": "COSINE",
           "params": {"ef": 64}},
    limit=10,
    expr='tenant == "acme" and year >= 2024',
)

A filter that removes most of the corpus is the hard case: the graph traversal visits many candidates that the filter then discards, so recall falls unless ef is raised. If one tenant’s data is a small fraction of a large collection, use partitions - Milvus can restrict the search to specific partitions, which is a partition-key strategy rather than a filter and keeps recall intact.

That is the multi-tenancy answer here: partition by tenant, do not rely on an expression filter alone. See Multi-tenant data isolation.

Choosing it, honestly

Situation Reach for
Under ~10M vectors pgvector, or Qdrant
Tens of millions Qdrant
Hundreds of millions Milvus

The operational cost is the deciding factor. Milvus distributed brings etcd, an object store and a message queue; that is justified by scale, GPU needs or disk-based indexes, and hard to justify below it. Running Milvus for two million vectors is a common over-engineering answer, and saying so is a better signal than praising it.

Interview angle 6

  • “Why would you choose Milvus?” - scale and hardware. Its layers - access, coordinator, worker, storage - scale independently, so re-indexing adds index nodes without touching query capacity, and it supports DiskANN for datasets beyond RAM plus GPU indexes. Below roughly a hundred million vectors that architecture is cost, not benefit.
  • “Why is a vector searchable but slow right after insert?” - it lands in a growing in-memory segment searched by brute force, and only becomes fast once the segment seals and is indexed.
  • “You insert then immediately query and get nothing. Why?” - the default Bounded consistency allows staleness within a window. Use Strong for read-after-write paths, accepting the latency, or the bug appears only under timing.
  • “How do you handle multi-tenancy?” - partition by tenant and restrict the search to partitions, rather than relying on an expression filter. A highly selective filter over a large collection makes the graph traverse many candidates that are then discarded, so recall drops unless ef is raised.
  • “Which index?” - HNSW as the default, IVF_PQ when memory is the constraint and some recall can be traded, DiskANN when the dataset exceeds RAM, GPU indexes when query volume justifies the hardware. The metric type must match the embedding model, since a mismatch degrades recall silently instead of erroring.
  • “What does it cost to run?” - a distributed deployment depends on etcd, object storage and a message broker. That is real operational surface, which is why Qdrant or pgvector is the better answer at smaller scale.
Previous
Qdrant in production