Time-series storage
Append-only metrics with compression, downsampling and retention.
01The intuition
The problem it solves
A hospital ward records each patient's temperature every hour. The readings are never edited, they arrive in time order, and the questions asked are always about a stretch of time: "the last 24 hours for bed 7", "average for the ward this week", "alert if anyone crosses 39 degrees". Filing each reading as a separate card in a general cabinet makes every question a long search and fills the cabinet within a month.
Servers produce the same kind of data at a far larger rate: CPU load, request counts, error rates, every few seconds from every machine and every service. A fleet of 100,000 machines with 200 metrics each sampled every 10 seconds is 2 million points a second, written once and read as ranges. A time-series store is built around those three facts: append-only, ordered by time, queried by range.
Build it one piece at a time
Step 1: name the series. A series is a metric name plus a set of labels: cpu_load{host="web-17", dc="bom"}. Each distinct combination of labels is its own series with its own id.
Step 2: append points in time order. Each series is a list of (timestamp, value). Writes append to the end; nothing is updated in place. Group the incoming points by series in memory and flush them in blocks of, say, two hours.
| Series | Raw points (16 bytes each) | Compressed |
|---|---|---|
| cpu_load{web-17} | (10:00:00, 0.42), (10:00:10, 0.43), (10:00:20, 0.43)… | timestamps as "every 10 s" deltas (a bit or two each), values as XOR of neighbours (a few bits each): about 1.4 bytes per point |
Step 3: compress by exploiting regularity. Timestamps are nearly evenly spaced, so store the difference of differences, which is usually zero. Values change little between samples, so store the XOR with the previous value, which has long runs of zero bits. Together these cut 16 bytes a point to about 1 to 2.
Step 4: index by series and time. An inverted index from each label value to the series ids containing it answers "all series with dc=bom" (inverted index). Each two-hour block is a file; a query for "last 6 hours of cpu_load for web-17" opens 3 blocks and reads one series from each.
Step 5: downsample and expire. Nobody looks at 10-second points from last year. A background job rolls old blocks into 1-minute and then 1-hour averages, minima and maxima, and deletes raw data past its retention (say 15 days raw, 13 months at 1 hour). Queries pick the resolution that matches the range.
Why it works, with the numbers
The write path is an append and the storage is compressed, so the fleet above is affordable:
2,000,000 points/s × 1.5 bytes ≈ 3 MB/s ≈ 260 GB/day raw; at 15 days raw plus hourly rollups for a year, under 5 TB
Reads are range scans over a few blocks, so a dashboard panel over 6 hours of one series touches a few kilobytes. The guarantee is that points arrive in time order per series and that a block, once written, is immutable, which makes replication and caching simple. The failure modes are cardinality explosions (a label like user_id creates a series per user, millions of series, and the index and memory blow up), out-of-order points from a machine with a wrong clock (accept a small window, drop the rest), and queries over many series at once (a regex over 10 million series is a scatter over all of them).
The trade-off you must say out loud
Time-series stores give up updates and arbitrary queries for cheap writes, compression and fast range reads. Say the retention and resolution policy, because it is where cost lives, and say which labels are allowed, because cardinality is where the store dies. The interviewer's follow-up is typically "how do alerts work?", and the answer is a rule engine that runs range queries every interval against the latest block, not a trigger on every write.
Where it shows up in the 100 Google problems
- Design a metrics and monitoring system: the store above, sharded by series, with alert rules on top.
- Design server health monitoring and failure detection: heartbeats and health signals are series whose absence is the alert.
- Design a service that shows the latest stock prices: ticks per symbol are a high-rate series; the latest value sits in a cache in front of the store.
- Design a real-time analytics pipeline: aggregated counts per minute per dimension are time series.
- Monitor an ML model's performance in real time: prediction distributions and accuracy over time, compared against a baseline series.