Caching strategies

Read-through, write-through, write-back, TTLs and invalidation.

System design conceptCaching and edges1 Nov 2026

01The intuition

The problem it solves

Picture a school library with one librarian. Every time a student asks for the class timetable, the librarian walks to the back room, finds the file, copies the page, and walks back. That walk takes two minutes. If 300 students ask on Monday morning, the librarian spends 600 minutes on one sheet of paper that has not changed since last week.

A web service has the same shape. The "back room" is a database on another machine: a lookup costs around 5 milliseconds plus a disk read, and the database can serve perhaps 5,000 such lookups a second. A popular page can be asked for 50,000 times a second. The database is ten times too slow, and most of those asks are for the same few answers.

The fix is the obvious one: keep a copy of the timetable on the front desk. That copy is a cache, which is a small, fast store of recent answers kept close to whoever asks. The hard part is not keeping the copy; it is knowing when the copy is wrong.

Build it one piece at a time

Step 1: a read-through cache. Put a fast memory store (Redis or Memcached, or just a map inside the server) in front of the database. A read first checks the cache. If the key is there, that is a hit and the answer returns in about 0.5 ms. If not, a miss, the service reads the database, stores the answer in the cache, and returns it. The next asker gets a hit.

Request	Cache before	Action	Cache after	Cost
timetable 7B	empty	miss, read DB, store	7B	5 ms
timetable 7B	7B	hit	7B	0.5 ms
timetable 8A	7B	miss, read DB, store	7B, 8A	5 ms

Step 2: a time to live. The teacher changes the timetable and the front-desk copy is now a lie. The simplest rule is an expiry: every cached entry carries a TTL, say 60 seconds, after which it is thrown away and the next reader refills it. Readers may see data up to 60 seconds stale, and that is a decision you make per key: a stock price gets 1 second, a user's profile photo gets a day.

Step 3: invalidation on write. When staleness is not acceptable, the code that writes to the database also deletes the cache entry. The next read misses and refills with the new value. Delete, not update: two writers racing to update the cache can leave the older value last, while two deletes are harmless.

Step 4: choose where writes go. Three patterns cover almost everything:

Write-around: writes go only to the database; the cache fills on the next read. Simple, good when written data is rarely read soon.
Write-through: writes go to the cache and the database together, so the cache is never stale, at the cost of a slower write.
Write-back: writes go to the cache only and are flushed to the database later in batches. Fastest writes, but a cache crash loses the unflushed ones. Used for counters and view counts, not for payments.

Step 5: eviction. Memory is finite. When the cache is full, drop the least recently used entry (LRU). A hot key stays because it keeps being touched; a key nobody asked for in an hour goes first.

Why it works, with the numbers

The quantity that matters is the hit rate h, the fraction of reads answered from the cache. Average latency is a weighted mix:

latency = h × 0.5 ms + (1 − h) × 5 ms

With h = 0.9 the average is 0.95 ms, five times better. The load on the database falls to (1 − h) of the original: 50,000 reads a second become 5,000, which the database can take. Hit rates of 90 to 99 percent are normal for read-heavy products, because popularity is uneven: a few thousand items get most of the traffic.

The guarantee a cache gives is bounded staleness: a reader sees a value that was true at most TTL seconds ago, or, with invalidation on write, a value that was true until the moment of the last write. The failure modes are the mirror image. A cache restart drops h to zero and the database sees full load for a few minutes. A missed invalidation serves a wrong answer until the TTL saves you. A popular key expiring sends thousands of readers to the database at once, which is the stampede problem with its own article.

The trade-off you must say out loud

You are trading freshness and memory for speed and database load. Say which keys may be stale and for how long, and which may not. The question that follows is always "what happens when the cache dies", and the honest answer is "the database must survive the miss storm, or we warm the cache before sending traffic". Also say whether the cache sits in each server (fast, but each server has its own copy and its own misses) or in a shared cluster (one copy, but a network hop).

Where it shows up in the 100 Google problems

URL shortener: 100 reads for every write, so a cache in front of the link table answers almost everything.
Social news feed: each user's assembled feed is cached so a refresh does not rebuild it.
Distributed cache: the cache itself as the product, with eviction and routing across machines.
Configuration service: every server caches flags locally and refreshes on a short TTL.
LLM serving: identical prompts are answered from a cache before a GPU is touched.