Concepts / Storage and data / Write-heavy workloads

Write-heavy workloads

Absorbing millions of writes a second with batching, append-only logs and partitioning.

System design conceptStorage and data1 Nov 2026

01The intuition

The problem it solves

A toll plaza on a highway logs every car: plate, lane, time. Ten lanes, a car every 2 seconds each, is 5 writes a second, fine for a clerk with a notebook. Now it is every toll plaza in the country and every sensor on every road: millions of events a second, each one tiny, none of them ever edited, and the readers want totals and recent history, not single events. The clerk's notebook does not scale; neither does a database that treats each event as a careful, indexed row.

Metrics, logs, trades, clicks and telemetry all arrive this way: far more writes than reads, append-only, time-ordered, high volume. A write-heavy design accepts them at line rate by never doing per-write random work, and pays the cost of organising them later and in bulk.

Build it one piece at a time

Step 1: measure the naive path. One row per event into a relational table with three indexes: each insert updates the table and three B-tree indexes, four random disk writes. A disk doing 10,000 random writes a second commits 2,500 events a second. You need 1,000,000.

Step 2: append, do not insert. Write events to the end of a log file in arrival order. Sequential writes run at disk bandwidth: 200 MB/s divided by 100-byte events is 2 million events a second on one disk. No index is touched on the write path.

After stepWrites/s on one machineWhat a read costs
1 (indexed rows)about 2,500index lookup, fast
2 (append-only log)about 2,000,000a scan, slow
3 (batched, partitioned, indexed later)millions across machinespartition pruning plus a small index, fast enough

Step 3: batch on the way in. Clients and collectors buffer events for a few hundred milliseconds and send them in batches of thousands. One network call and one disk write per batch instead of per event. A durable queue in front absorbs bursts and lets writers and storage scale separately.

Step 4: partition by time and key. Each hour (or each 1 GB) becomes its own file, and the files are spread across machines by a key such as source or region (sharding). A query for "errors from region X in the last hour" opens one file, not all of them.

Step 5: build indexes and summaries later. Once a partition is closed it is immutable; a background job sorts it, builds a small index and computes rollups (counts per minute, per key). This is the LSM idea applied to a whole pipeline: cheap writes now, organisation in the background, reads that are fast enough because they touch few partitions.

Why it works, with the numbers

Everything comes from the sequential-versus-random gap and from batching:

cost per event ≈ (cost per batch) ÷ (events per batch); a 1 ms disk write amortised over 5,000 events is 0.2 µs per event

Partitioning by time gives free expiry (delete the oldest file) and makes compaction local. The guarantee is that an acknowledged event is in the log and will appear in queries within the organisation delay (seconds to a minute). The failure modes are batching that loses the last batch in a crash (keep a write-ahead log or require the queue to persist before acknowledging), a hot partition when the key is skewed (one source producing half the events), and background jobs that fall behind so reads start scanning raw files.

The trade-off you must say out loud

You trade read latency and immediacy for write throughput: queries see data a little late and read from partitions rather than indexes. Say the batch window, the partition size and the acknowledged-but-not-yet-queryable delay. The interviewer will ask "what if I need to read a single event by id right now?", and the answer is a small hot store (the last few minutes in memory) in front of the partitioned files.

Where it shows up in the 100 Google problems

02Used in these problems