What are embeddings

Article.

A dense vector that positions text in a space where geometric closeness approximates semantic similarity. That one property turns “find things that mean the same” into a nearest-neighbour lookup, which is a solved engineering problem.

"cancel my order"    -> [0.12, -0.44, ...]
"how do I refund"    -> [0.15, -0.41, ...]  close
"reset my password"  -> [-0.61, 0.22, ...]  far

The numbers mean nothing individually. Only distances mean anything, and only within one model’s space.

The problem this solves has a name — vocabulary mismatch. The question says “linter”, the document says “lint”. The user asks to “cancel”, the policy says “refund”. Keyword search scores those at zero.

The mechanism: what “similar” was trained to mean

Embeddings are not a discovered property of language. They are the output of an objective someone chose, and knowing which objective tells you what the vectors will and will not encode.

Modern text embedding models are trained contrastively: show the model a query and its matching passage, pull those two vectors together, and push the query away from every other passage in the batch.

batch of 4 pairs:

  q1 ── pulled toward ──> p1
   └─── pushed from ────> p2, p3, p4   (in-batch negatives)

Repeat over hundreds of millions of pairs and the geometry falls out: things that co-occur as query-and-answer end up close.

Two consequences that matter more than the training detail:

• “Similar” means similar in the training distribution’s sense. A model trained on web search pairs learns web-search relevance. Ask it whether two contract clauses are equivalent and you are outside what it was optimised for.
• Hard negatives are what make a model good. Random negatives are easy — a query about refunds is trivially far from a recipe. Progress comes from training against passages that look right and are not, which is also why domain fine-tuning works so well when it works.

The antonym problem

Here is where the mechanism bites, and it surprises people.

Contrastive training inherits the distributional assumption: text appearing in similar contexts gets similar vectors. Antonyms appear in almost identical contexts.

sim("the price increased sharply", "the price decreased sharply")   # ~0.95
sim("the price increased sharply", "the weather was pleasant")      # ~0.10

Opposite meaning, near-identical vectors. And it generalises to negation, which embedding models capture poorly:

query:  "how do I cancel my subscription"
chunk:  "Subscriptions cannot be cancelled after trial."
        -> retrieved with high confidence

That chunk is topically perfect and answers the opposite of what was asked. Vector search alone cannot see the not, because “can” and “cannot” sit in overwhelmingly similar contexts.

This is a real production failure mode, not a curiosity — it is why a reranker matters, since a cross-encoder reads query and passage together and can attend to the negation. See Hybrid search and reranking.

Static versus contextual

Static embeddings gave one vector per word regardless of use, so bank was an average of two unrelated meanings. Word2Vec is worth knowing as history — the king - man + woman ≈ queen arithmetic is where the intuition comes from — but proposing it for retrieval today dates you.

A modern embedding model also produces one vector per input, pooled from the token vectors. When someone says “embedding model” for RAG, they mean that pooled whole-passage vector, not per-token ones.

Similarity: normalise once

# Normalise at index time, not per query.
v = v / np.linalg.norm(v, axis=1, keepdims=True)
q = q / np.linalg.norm(q)

# A plain dot product is now exactly cosine, as one matrix multiply.
scores = v @ q

Raw dot product mixes direction (semantics) with magnitude, and magnitude often tracks passage length rather than meaning — so a long, vaguely related chunk outranks a short exact one. Normalising removes that and makes dot product and cosine identical.

Some vector databases default to Euclidean. If relevance looks subtly wrong and long documents keep winning, check the index metric before you change models.

Dimensions cost money

Dimension count drives storage, memory and query latency roughly linearly. A million chunks at 1536 float32 dimensions is about 6 GB before index overhead; at 384 it is 1.5 GB.

Models trained with Matryoshka representation learning let you truncate the vector and keep most of the quality — take 512 dimensions from a 3072 model and renormalise, rather than switching to a weaker model you have not evaluated.

And try quantisation first: int8 instead of float32 cuts memory 4× for a small recall loss, binary goes further with rescoring. Both keep the model whose quality you already measured.

Choosing a model

On your own retrieval eval, not a leaderboard. Public benchmark scores are averaged over tasks that are probably not your task, and benchmarks leak into training sets.

MTEB and BEIR, and how to read them

BEIR is a zero-shot retrieval benchmark — a set of retrieval tasks a model was not trained on, which is the property that makes it informative. MTEB is the broader successor: retrieval plus classification, clustering, reranking, semantic similarity and more, with a public leaderboard that is where most teams start.

Use them to build a shortlist of three or four candidates, then decide on your own data. Four reasons the leaderboard order is not your order:

• The average hides your task. MTEB’s headline number averages across task types. If you only do retrieval, read the retrieval column; a model can rank well overall and mediocrely at the one thing you need.
• Contamination. The leaderboard is public and the datasets are public, so later models have had the opportunity to train on them. Zero-shot claims weaken over time.
• Domain. BEIR and MTEB are broad web and academic text. Legal, clinical and code corpora reorder the table.
• Optimisation to the target. A benchmark that decides adoption gets optimised for, which is a general property of benchmarks and not a scandal — it just means the gap to your data is wider than the gap between rows.

The practical rule: shortlist from MTEB’s retrieval task type, filter by max input length and dimension, then run recall@k on your own queries. A model two places lower on the leaderboard that handles your chunk size without truncating will beat the leader.

In priority order:

1. Recall@k on 30 of your real queries. Nothing else predicts production.
2. Max input length against your chunk size. A 512-token model silently truncates a 900-token chunk and the tail is simply not indexed.
3. Dimension, for the cost reasons above.
4. Multilingual, if the corpus is.
5. Where it runs. An API model means documents leave your boundary, which is decided for you in regulated work.

Asymmetric retrieval needs its prefixes

Retrieval is asymmetric — a short question must match a long passage — and many models are trained with explicit prefixes for the two roles:

q = model.encode("query: how do I cancel")
p = model.encode("passage: To cancel, open Settings > Billing...")

Omit them and you lose real accuracy with no error and no warning. Check the model card; this is one of the most common silent misuses in the field.

Where it breaks in production

Changing the model. Vectors from two models are not comparable and there is no conversion. A model change is a full re-embed and re-index, and until it completes, mixed vectors produce meaningless distances. Version the index, build the new one alongside, cut over. On a large corpus that is hours of compute and a migration, not a config change.

Silent truncation. Chunks longer than the model’s limit are cut, not rejected. The tail of every long chunk is unsearchable and nothing tells you. Assert chunk length against the model limit at ingest.

Missing prefixes, as above — a quality regression with no failure signal.

Negation and antonyms, as above — mitigate with a reranker, and with hybrid retrieval so exact terms carry weight.

Embedding the wrong text. The body alone is often less discriminative than the body with its section path prepended. What you embed need not be what you display.

The trade not taken

Embeddings are not the only way to find text, and reaching for them first is a habit worth questioning.

BM25 needs no model, no GPU, no re-index on upgrade, and beats dense retrieval outright on exact identifiers — error codes, SKUs, proper nouns. It is a strong baseline, it is cheap, and a system that starts there and adds vectors where they demonstrably help is easier to reason about than one that starts with embeddings and discovers the code-lookup queries later.

The production answer is usually both, which is the honest version of “use embeddings”.

Related

• Vector Databases - how nearest-neighbour search is actually done
• Chunking - what you embed matters more than which model
• Fine-tuning embeddings - adapting a model to your domain