Design a recommendation system

Candidate generation, ranking and re-ranking for YouTube or a feed, under a 200 ms budget.

Problem 092 of 100ML and AI systemsL51 Nov 2026

Concepts this design leans on

Recommendation system architecture — Candidate generation, ranking and re-ranking under a latency budget.
Embeddings and nearest-neighbour search — Turning items into vectors and finding similar ones in milliseconds.
Feature stores — The same features at training time and serving time.
Ranking pipelines — Scoring thousands of candidates in stages, cheap first, expensive last.
A/B testing and experimentation — Assigning users consistently and measuring a difference you can trust.

01The design, in the six-step framework

1. Requirements

Functional.

Given a user opening the home feed (YouTube, a shopping app), return an ordered list of items they are likely to engage with, personalised from their history and context.
New items (uploaded a minute ago) and new users (no history) must get reasonable recommendations.
Engagement (clicks, watch time, purchases) feeds back into the models; experiments compare ranking changes.
Out of scope: the content catalogue itself, content moderation, the UI.

Non-functional.

100 million daily users, 50,000 feed requests per second at peak, from a catalogue of 1 billion items.
End-to-end latency budget 200 ms at p99 for the whole page, of which recommendation gets 120 ms.
Freshness: a user's action influences their next request within minutes; a new item is recommendable within an hour.
Availability 99.9 percent, with a degraded (popular-items) fallback when models are unavailable.

Back of the envelope

Scoring 1 billion items per request is impossible: at 1 microsecond per item that is 1,000 seconds. The budget of 120 ms allows ranking perhaps 1,000 candidates with a heavy model (100 microseconds each) and 10,000 with a light one.

Candidate generation must cut 10⁹ → 10³ in about 20 ms; approximate nearest-neighbour search over item embeddings does that in single-digit milliseconds

Embeddings: 1 billion items × 128 floats × 2 bytes = 256 GB, sharded across an ANN cluster. User features for 100 million users at 2 KB are 200 GB in a key-value store. Training data: 50,000 requests/s × 20 impressions = 1 million impressions per second logged.

2. Core entities

User profile: user_id, long-term features (topic affinities, embedding), recent history (last 50 interactions with timestamps), context (device, locale, time).
Item: item_id, metadata, embedding, quality and freshness features, aggregate statistics (impressions, CTR per segment).
Candidate set: request_id, sources (ANN, co-visitation, subscriptions, trending), item ids with source scores.
Impression log: request_id, user, item, position, score, features snapshot, model version.
Engagement event: user, item, type, value, timestamp; joined with impressions to make labels.

3. API

GET /recommendations?user_id=&surface=home&count=20&cursor= → ranked items with a request_id and per-item reason codes. Called by the app; must return within budget or fall back.
POST /events {user, item, type, request_id} → 202; the feedback path, fire-and-forget from the client.
Internal: POST /candidates {user features} → item ids from each source; POST /score {user features, items} → scores; both with deadlines.
GET /items/{id}/embedding and the batch job interfaces that refresh embeddings and statistics.

4. High-level design

One feed request:

The orchestrator fetches the user's features from the online feature store: profile, embedding, last 50 interactions (updated by a stream job within seconds of an event).
It fans out to candidate generators in parallel with a 25 ms deadline: ANN search for items near the user embedding, co-visitation lists for the last few items watched, subscriptions or follows, trending-in-region, and a fresh-items source; each returns a few hundred ids, deduplicated to about 1,000.
The ranker scores all candidates in one batched call to the model servers (60 ms deadline): features are assembled per (user, item) pair and a model predicts engagement probabilities.
The re-ranker applies business rules and diversity (no three videos from one channel, interleave fresh items, demote recently shown), then takes the top 20.
Impressions are logged with the features used and the model version; engagement events arrive later and are joined to make training labels.

5. Deep dives

Why a funnel, and how is each stage sized?

Because cost per item and quality per item trade off: the cheap stages can look at millions, the expensive stage can only afford hundreds. The funnel is candidate generation (recall: find everything that could be good), ranking (precision: order it), re-ranking (policy: make the list good as a whole). Sizing comes from the budget: with 60 ms for ranking and model servers that score a batch of 1,000 in 40 ms, 1,000 candidates is the ceiling; a lighter first-stage ranker could pre-score 10,000 to pick those 1,000 if recall is the problem. The orchestrator enforces deadlines per stage and proceeds with whatever candidates returned in time; a generator that is slow degrades quality, never availability.

How do new items and new users get handled (cold start)?

A new item has no engagement statistics and an embedding only from its content (title, thumbnail, audio, uploader), which is enough for the ANN index after a batch job embeds it, within the hour. A dedicated fresh-items candidate source guarantees it some impressions through an exploration slot in the re-ranker (say 5 percent of positions), and its statistics are updated by the stream job as events arrive; after a few hundred impressions the ranker has real signal. A new user has context (country, device, language, the item they arrived from) and that is the whole profile; the generators that work are trending, popular-in-segment and co-visitation from the first item watched. The system is designed so that both cases are ordinary candidate sources with their own weighting, not special code paths.

Stage	Input size	Model	Deadline	Serving
Candidate generation	10⁹	Two-tower embeddings, heuristics	25 ms	ANN index, key-value lists
Ranking	About 1,000	Deep model with cross features	60 ms	GPU or CPU model servers, batched
Re-ranking	About 100	Rules, diversity, calibration	10 ms	In the orchestrator

How is training data kept honest, and features consistent between training and serving?

The impression log stores the exact feature values the ranker saw at request time; training joins labels onto that snapshot. Computing features afresh at training time from today's data would leak the future (the item's CTR after it went viral) and skew the model, the classic training-serving skew. The feature store serves the same feature definitions online (low latency, keyed by id) and offline (point-in-time correct historical values) from one pipeline, which is what the term means in practice. Position bias is the other trap: items shown at the top get clicked more, so the model is trained with position as a feature and served with position fixed, or the labels are weighted by inverse propensity.

How do we know a new model is better, and ship it safely?

Offline metrics (AUC on held-out impressions) rank candidates for a test; online A/B tests on the real metric (watch time, retention) decide. The model registry holds versions; the ranker loads a version by name; the experiment platform assigns users to versions and the impression log carries the version so results are attributable. Rollout is by cohort with automatic rollback on a drop in engagement or a rise in latency. Feedback loops are the long-term risk: a model that only recommends what it already showed narrows the catalogue; the exploration slot and diversity rules are the counterweight and should be measured as their own metric.

6. What a strong answer sounds like

Open with the arithmetic that makes a funnel mandatory: a billion items, a 120 ms budget, a thousand candidates.
List candidate sources plural, with ANN as one of them, and say the orchestrator runs them in parallel under deadlines.
Explain the impression log with feature snapshots and name training-serving skew and position bias.
Treat cold start as sources with weights, not an afterthought.
Expected follow-ups: how embeddings are trained (two-tower on co-engagement); how to update the ANN index (periodic rebuild plus a small fresh index merged at query time); how to keep latency when the feature store is slow (cached profiles, deadline and fallback).
Common mistake: describing one model that scores everything, or forgetting that the candidate stage, not the model, decides what can ever be recommended.

02Where this question was reported

DesignGurus · Relevant to YouTube, Google News, Play Store
DesignGurus · Listed
Exponent · ML SD: 'recommendation and ranking system for a feed at billions of users'
System Design Handbook · Listed
scaleengineer.com · Asia: 'recommendation engine for an e-commerce platform'
scaleengineer.com · 'build a recommendation engine' from a user-interaction dataset
Taro · Design a recommendation engine for a social media platform; loop also had a separate ML system design round
Blind · Recommendation system for e-commerce; interviewer stayed on inputs/outputs and product value, never reached architecture (felt like a PM question)
1point3acres · ML design: video recommendation system and an Explore-style recommender (search snippet, Chinese)
Prepfully · Candidate reports: recommendation and ranking for a feed at billions of users; ML probes on retrieval/ranking justification, label noise, position bias, cold st
PracHub · Dec 2025 MLE, rejected: real-time recommendation; interviewer pressed latency vs accuracy; weak cold-start and monitoring answers.
grandavehousing.calpoly.edu · 'Design a recommendation system for YouTube' with data, features, model, eval, scale.