Recommender systems

The classic ML system-design question, and the one where candidates most often describe a model when the interviewer wanted an architecture.

The answer that lands: recommendation is a two-stage funnel, not a prediction.

millions of items
      │  retrieval        (fast, approximate)
      ▼
   ~500 candidates
      │  ranking          (slow, precise)
      ▼
    ~20 shown
      │  re-rank: diversity, business rules
      ▼
     the feed

You cannot score ten million items per request in 50ms. So stage one throws away 99.99% cheaply, and stage two spends real compute on what survived. Every production recommender has this shape.

Retrieval: the cheap stage

Collaborative filtering learns from the interaction matrix: users who agreed before will agree again. It needs no item features and it cannot say anything about an item nobody has touched.

Two-tower is the modern default: one encoder for the user, one for the item, trained so the dot product of their embeddings predicts interaction. Item embeddings are precomputed into an ANN index, so serving is one user embedding plus a nearest-neighbour lookup — the same machinery as Vector Databases.

In practice you run several retrievers and union the results: collaborative for the familiar, content-based for the new, popularity as a floor.

Ranking: the expensive stage

With 500 candidates you can afford real features — user history, item attributes, context, cross-features — and gradient boosting or a neural ranker.

The framing that matters: rank by expected value, not by predicted click.

score = p(click) x value(item)

Optimising click alone gives you clickbait, because the model finds what gets clicked rather than what is worth showing. Real systems combine several predicted outcomes — click, purchase, dwell time, return next week — weighted by what the business actually wants.

Cold start is the real problem

Three flavours, and they have different answers:

• New user — no history. Fall back to popularity, ask for preferences during onboarding, or use whatever context you have (device, region, referrer).
• New item — nobody has interacted. Content-based retrieval carries it until it has signal, and an explicit exploration budget gets it seen.
• New system — no interactions at all. You are building a rules engine first, and that is the correct answer rather than an admission.

The exploration point is the one that separates a considered answer. A recommender trained only on what it showed learns from its own output — a feedback loop that narrows the catalogue over time. Reserving a small slice of traffic for exploration is how you keep the training data honest, and bandits are the principled version.

Evaluation, and the trap in it

Offline metrics disagree with online ones, routinely and by design. Offline evaluation replays what users did given what they were shown, so it cannot tell you what would have happened had you shown something else. A recommender that scores better offline can lose an A/B test.

Consequences worth stating:

1. Offline is a filter for obviously worse models, not a decision procedure.
2. Ship behind an A/B test and measure the business metric.
3. Watch coverage and diversity alongside accuracy, or you will optimise into a system that recommends the same twenty items to everyone.

Position bias makes this worse: users click the top result because it is on top. Training on raw clicks bakes that in, and correcting for it — inverse propensity weighting, or randomised positions on a slice of traffic — is an advanced answer worth having.

Serving shape

request ──▶ feature store (user + context)
        ──▶ ANN retrieval (precomputed items)
        ──▶ ranker
        ──▶ business rules, dedupe, diversity

Item embeddings are refreshed in batch, hourly or daily. The user vector is either precomputed too, or built at request time from recent events — which is the latency/freshness trade-off, and where a Feature stores and point-in-time correctness earns its place.

The last box is not an afterthought. Deduplication, “don’t show what they bought yesterday”, supplier quotas and legal exclusions live there, and they override the model.

Related

• Dimensionality reduction — matrix factorisation is the ancestor
• Vector Databases — the ANN index under retrieval
• Online evaluation and experiments — A/B testing the change
• ML system design framework — how to structure the whole answer