Fan-out on write versus fan-out on read
Pushing a post to every follower's inbox, or assembling the feed when asked.
01The intuition
The problem it solves
A class group has 40 students. When the teacher posts "test tomorrow", there are two ways the notice can reach everyone. The teacher can write it on 40 slips and put one in each student's locker. Or the teacher can pin it on one board, and each student checks the board when they want. The first way costs the teacher 40 writes but a student reads her own locker in one glance. The second costs the teacher one write but every student must look at every board for every teacher they follow.
A social feed is exactly this with bigger numbers. A user follows 300 accounts. Opening the app should show the latest posts from all 300, sorted by time, in under 200 ms. Either the posts were delivered to her in advance, or they are gathered when she asks. Both work. Neither works for everyone.
Build it one piece at a time
Step 1: fan-out on read, the board. Store each post once, under its author. When a user opens the app, fetch the latest 20 posts from each of her 300 followees, merge them by time, return the top 20. That is 300 lookups per app open. At 10,000 app opens a second it is 3 million lookups a second. Writes are cheap, reads are expensive.
Step 2: fan-out on write, the lockers. Keep an inbox list per user, in a fast store. When someone posts, look up their followers and append the post ID to each follower's inbox. Opening the app reads one list. At 10,000 app opens a second that is 10,000 reads. Reads are cheap, writes are expensive.
| Event | Posts table | Inbox of user A (follows T and S) | Inbox of user B (follows T) |
|---|---|---|---|
| T posts p1 | p1 by T | [p1] | [p1] |
| S posts p2 | p1, p2 | [p2, p1] | [p1] |
| A opens app | unchanged | read [p2, p1], fetch the two posts | unchanged |
Step 3: hit the celebrity problem. A singer with 50 million followers posts once. Fan-out on write must append to 50 million inboxes. At 100,000 appends a second that is over 8 minutes, during which followers see nothing, and the write storm starves everyone else's writes.
Step 4: the hybrid. Fan out on write for ordinary accounts, say under 10,000 followers. For celebrities, do not fan out; instead, when a user opens the app, read her inbox and also fetch the latest posts of the few celebrities she follows, then merge. Most users follow a handful of celebrities, so the read-time work is small, and the write storm is gone.
Step 5: keep inboxes bounded. An inbox holds the last 500 post IDs, not the full history. Users who have not opened the app in a month get no inbox at all; their feed is built on read when they return. That removes most of the write cost for the long tail of inactive accounts.
Why it works, with the numbers
Let F be the average number of followers and R the reads per post. Fan-out on write costs F writes per post; fan-out on read costs about R × (accounts followed) lookups. With 300 followees, 10 opens per user per day and posts read by 100 users on average:
write cost per post = F = 300; read cost per post without inboxes ≈ 100 reads × 300 followees / posts per followee ≈ thousands
So for ordinary accounts the inbox wins by about an order of magnitude, and the hybrid caps the write cost at the follower threshold. The invariant to keep: every post appears in a follower's feed within a few seconds, either because it was pushed or because it is fetched on read. Failure modes: a lost inbox append (the post is missing for one follower until a repair job runs), and stale inboxes after an unfollow or a deleted post, which need a cleanup step.
The trade-off you must say out loud
Write amplification versus read latency, and storage for inboxes versus compute at read time. Say the follower threshold where you switch, how inactive users are handled, and how deletes propagate. The follow-up is "what about ranking", and the answer is that the inbox gives you candidates, and a ranking step reorders the top few hundred on read.
Where it shows up in the 100 Google problems
- Social news feed: the hybrid is the core of the design, with the celebrity case as the first deep dive.
- Instagram: the same inboxes, with post IDs pointing at media on a CDN.
- YouTube live chat: one message to a million viewers is fan-out on write at its most extreme, done through pub/sub, not inboxes.
- Latest stock prices: a tick is fanned out to every subscriber's open connection.