Recommendation system architecture
Candidate generation, ranking and re-ranking under a latency budget.
01The intuition
The problem it solves
YouTube has about a billion videos. When Asha opens the app, it must show her twenty she will probably watch, and it must do so in about 200 ms. The honest way to pick the best twenty is to score every video for Asha with a model that looks at her history and the video's features. A good model takes about a millisecond per video on a fast machine:
109 videos × 1 ms = 106 s ≈ 11 days per page load
So the best model cannot see every video. The design question is how to spend a 200 ms budget: look at a lot of videos cheaply, then a few expensively.
Build it one piece at a time
Step 1: candidate generation, from a billion to a few thousand. Several cheap sources each propose candidates in a few milliseconds: videos from channels Asha subscribes to; videos watched by people who watched what she watched (collaborative filtering); videos whose embedding is near the embedding of her recent history; what is trending in her region. Each returns a few hundred ids. The union is about 2,000 candidates.
Step 2: filter. Remove what she has already watched, what is unavailable in her country, what is flagged, and what she said she does not want. Cheap set lookups; 2,000 becomes 1,500.
Step 3: ranking, from 1,500 to 50. A heavier model scores each remaining candidate using features of Asha, the video and the context (time of day, device). It predicts, for example, the probability she clicks and the expected watch time. At 0.05 ms per candidate with batching on an accelerator, 1,500 candidates take about 75 ms.
| Stage | Input | Output | Cost per item | Time |
|---|---|---|---|---|
| candidate generation | 109 | 2,000 | index lookups | 20 ms |
| filter | 2,000 | 1,500 | set lookups | 5 ms |
| ranking | 1,500 | 50 | 0.05 ms | 75 ms |
| re-ranking | 50 | 20 | rules | 5 ms |
Step 4: re-ranking, from 50 to 20. The top 50 by score may be ten videos from one channel. Rules enforce variety (at most two per channel), freshness (one video under a day old), and business constraints, and pick the final twenty in order.
Step 5: close the loop. Log what was shown and what Asha did. Those logs train the next version of the candidate and ranking models every day, and an experiment decides whether the new model replaces the old. Features used at serving time must be the same ones used in training, which is what a feature store guarantees.
Why it works, with the numbers
The funnel multiplies cheap breadth by expensive depth. Total cost is the sum of each stage's size times its per-item cost, and the stages are sized so each is a similar share of the budget:
20 + 5 + 75 + 5 = 105 ms of the 200 ms budget, leaving room for network hops and the page render
The guarantee is a latency bound with a quality that improves as models improve, never "the best twenty videos", since the best one might not have been a candidate. That is the honest limit: candidate generation decides what the ranker can ever choose, so a video no source proposes is invisible no matter how good the ranker is, and the loop can narrow a user's world because it mostly proposes things like what they already watched. Freshness is the other limit: features computed last night do not know what Asha watched this morning, which is why some features are updated in near real time from the event stream.
The trade-off you must say out loud
More candidates cost more ranking time; a bigger ranker costs more per candidate. Say the budget split. The interviewer's follow-up is the cold start: a new user has no history and a new video has no watches, so candidate generation falls back to popularity and content features until signals arrive.
Where it shows up in the 100 Google problems
- Design a recommendation system: this funnel, with the candidate sources, the two-tower retrieval model and the daily training loop made explicit.
- Design TikTok and a social news feed: the same funnel where the candidates are recent posts and the ranker predicts engagement.
- Design an ad-serving system: candidates are eligible ads, the ranker predicts click probability, and an auction replaces the re-ranking rules.
- Design Google News: stories instead of videos, with clustering before ranking.