Concepts / ML and AI serving / Ranking pipelines

Ranking pipelines

Scoring thousands of candidates in stages, cheap first, expensive last.

System design conceptML and AI serving1 Nov 2026

01The intuition

The problem it solves

A query to Google Search matches millions of pages. A user's home feed could be filled from tens of thousands of recent posts. An ad slot has thousands of eligible ads. In each case the system must put the best few at the top, and "best" is a judgement that takes real computation: a modern ranking model reads dozens of features and a chunk of text per item. Running it on a million items takes far longer than the 200 ms a user will wait.

The way out is the same trick a hiring committee uses. Nobody reads a thousand applications carefully. Someone screens them in a minute each, a smaller group gets a proper read, and three come to interview. Each stage is more expensive per candidate and sees fewer candidates.

Build it one piece at a time

Step 1: retrieval, cheap and broad. An index returns everything that could match: pages containing the query words, posts from followed accounts, ads targeting this keyword. No model yet, only lookups. Output: a few thousand items.

Step 2: a first-stage scorer, tiny and fast. A small model or a formula scores each item in microseconds using a handful of precomputed signals: text match strength, popularity, recency, the author's history. Keep the top 500.

Step 3: a second-stage scorer, heavy and accurate. A large model scores the 500 with hundreds of features, including features of the user and the pairing of user and item. It predicts what the product cares about: probability of a click, expected watch time, chance of a purchase. Keep the top 50.

StageCandidates inCost per itemStage timeCandidates out
retrieval106 (index)lookup15 ms3,000
first scorer3,0005 µs15 ms500
second scorer500100 µs50 ms50
re-rank50rules5 ms10

Step 4: re-rank with rules the model does not know. Diversity (not five results from one site), freshness, policy (nothing flagged), and for ads an auction where bid × predicted click rate decides the winner and the price. Output: the ten shown.

Step 5: fit it in the budget with deadlines. Each stage has a time limit. If the second scorer has only finished 400 of 500 when its 50 ms are up, the ranking continues with those 400. A slightly worse page on time beats a perfect page late.

Why it works, with the numbers

Total cost is the sum over stages of candidates times cost per candidate. The funnel keeps each stage's product roughly equal, so no stage dominates and the expensive model only ever sees a few hundred items:

3,000 × 5 µs + 500 × 100 µs + 50 × rules ≈ 15 + 50 + 5 = 70 ms, against 106 × 100 µs = 100 s for the heavy model on everything

The guarantee is a bounded latency and a quality that is close to running the heavy model on everything, as long as the cheap stage rarely drops items the heavy stage would have ranked top. That is the honest limit: the first stage is a filter trained to agree with the second, and when they disagree the good item is gone before anyone expensive looks at it. Measuring "recall of the final top ten at each stage" on logged traffic is how teams know whether the funnel is too narrow. The other limit is feedback: what is shown is what gets clicked, which trains tomorrow's model to show it again, so exploration (showing a few unproven items) is part of the design.

The trade-off you must say out loud

Wider stages cost latency and compute; narrower stages lose good candidates. Say the sizes and the per-stage budgets. The interviewer's follow-up is how a new, unseen item gets a chance (exploration and content features), and how the pipeline degrades when the heavy scorer is slow (serve first-stage order and log it as degraded).

Where it shows up in the 100 Google problems

02Used in these problems