TTLs and expiry

01The intuition

The problem it solves

A story on Instagram or WhatsApp lives for exactly 24 hours and then disappears. A login session should end after 30 days of inactivity. A cached price should be forgotten after 5 minutes. All three are the same requirement: data with a time to live (TTL), after which it must not be served.

The obvious way is a cleaner that runs every minute, scans every record, and deletes the expired ones. With 500 million stories posted a day, the table holds about 500 million live rows plus whatever has not been cleaned yet. Scanning half a billion rows every minute to find the few thousand that expired in that minute is almost all wasted work, and the scan competes with real traffic for the same disks.

Build it one piece at a time

Step 1: store the expiry time, not a countdown. Each record gets a field expires_at, set at write time to now plus the TTL. A countdown would need updating; a timestamp never changes.

Step 2: check on read. When a request asks for a story, the server compares expires_at with the current time and refuses to return an expired one, even if it is still on disk. From the user's point of view expiry is now exact to the second, regardless of when the cleaner runs. The cleaner is only about reclaiming space.

Step 3: index by expiry so the cleaner reads only what it needs. Add an index on expires_at. The cleaner asks "rows with expires_at before now", which the index answers directly: a few thousand rows, not half a billion. Delete them in small batches so the deletes do not lock the table.

Step 4: partition by time and drop whole partitions. Even better for data that all expires at the same age: store stories in one partition per hour of expiry. When the hour passes, drop the partition, which is one metadata operation instead of millions of row deletes. Object storage buckets do the same with lifecycle rules: "delete objects older than 24 hours".

Step 5: let the store do it. In-memory stores such as Redis take an expiry with each key and remove it lazily on access plus a small random sample in the background. Wide-column stores such as Bigtable and Cassandra attach a TTL per cell and drop expired cells during compaction, so expiry is free at write time and paid during work the store was doing anyway.

Why it works, with the numbers

Correctness comes from the read-time check, which is O(1) per request. Reclamation cost drops from a full scan to the size of the expired set:

full scan: 5 × 10⁸ rows per minute; indexed cleaner: 5 × 10⁸ ÷ 1,440 minutes ≈ 350,000 rows per minute; partition drop: 1 operation per hour

The guarantee is "nothing is served after its expiry" and, separately, "space is reclaimed within some lag", and it is worth saying both because they are different promises. The honest failure modes: clocks differ between servers, so a story might vanish a few seconds early on one server and late on another, which is fine for stories and not fine for a legal retention rule; and a replica that lags may still serve an expired row unless it also checks expires_at, which is why the check lives in the read path and not only in the cleaner.

The trade-off you must say out loud

Exact expiry on read is cheap; exact deletion on time is not, and usually not needed. Say which one the requirement really is. The interviewer may ask about data that must be gone for privacy reasons, where "not served" is not enough and the deletion lag becomes a number you must commit to, including in backups.

Where it shows up in the 100 Google problems

• Design the stories feature: expires_at on every story, read-time filtering, hourly partitions dropped whole, and media in a bucket with a 24-hour lifecycle rule.
• Design a distributed cache: per-key TTLs with lazy expiry, and the stampede that an expiring hot key can cause.
• Design a URL shortener: optional link expiry handled the same way, with the redirect path doing the check.
• Presence uses expiring keys so "online" is simply "the key still exists".

02Used in these problems

• Design the stories feature (24-hour posts)