Design a metrics and monitoring system

01The design, in the six-step framework

1. Requirements

Functional.

• Services and hosts report numeric metrics (request count, latency histograms, CPU, queue depth) as time series identified by a name and labels.
• Dashboards query ranges with aggregation (sum by region, p99 over 5 minutes) over the last hour to the last year.
• Alert rules evaluate continuously and fire within a minute of a condition becoming true.
• Out of scope: logs and traces, on-call paging workflow beyond emitting the alert.

Non-functional.

• 10 million active time series, each a sample every 10 seconds: 1 million samples per second.
• Alert latency: under 60 seconds from sample to notification.
• Query latency: under 1 second for a one-hour dashboard panel; under 10 seconds for a one-year panel.
• Retention: raw for 15 days, 1-minute rollups for 1 year, 1-hour rollups for 5 years.
• Availability: the monitoring system must stay up when the thing it monitors is down; no shared dependencies with production.

2. Core entities

• Series: series_id (hash of name + sorted labels), name, labels, first_seen, last_seen.
• Sample: series_id, timestamp, value; stored in compressed chunks of about two hours.
• Chunk: series_id, start, end, compressed bytes, resolution (raw, 1m, 1h).
• Alert rule: expression, duration ("for 5 minutes"), labels, receivers, state per series (inactive, pending, firing).
• Dashboard panel: query expression, range, step; cached results keyed by (expression, range, step).

3. API

• Ingest: POST /write batch of {series labels, samples}, or a pull model where the server scrapes GET /metrics on each target every 10 seconds.
• GET /query_range?expr=&start=&end=&step= → matrix of series with values; used by dashboards.
• GET /series?match= → label discovery for the query editor.
• PUT /rules/{id} {expr, for, severity, receivers} → rule registered with the evaluators.
• Outbound: POST receiver webhook {alert, labels, state} with an idempotent alert fingerprint.

4. High-level design

A sample's path and an alert's path:

1. A collector scrapes a target (or receives a push), attaches labels like region and job, and appends samples to the partitioned log, keyed by series hash so one series always lands on one ingester.
2. The ingester appends to a write-ahead log on disk, then into an in-memory head block per series, where the last two hours live uncompressed enough to append in O(1).
3. Every two hours the head is sealed into an immutable compressed block and shipped to object storage; the compactor later merges blocks and produces 1-minute and 1-hour rollups.
4. A dashboard query is parsed into series selectors plus an aggregation; the engine reads recent data from ingesters and older data from blocks, aggregates, and returns a matrix at the requested step.
5. Rule evaluators run every 15 seconds: they run the rule's query over the recent window, track per-series state, and when a condition has held for the rule's duration they emit a notification with a fingerprint so a retry does not page twice.

5. Deep dives

Why a time-series database, and what is inside it?

The interviewer from the reports asked exactly this, and the answer is the access pattern: writes are append-only, always at the newest timestamp, across millions of independent series; reads are range scans over one series or a labelled group. A general row store pays a B-tree insert per sample and stores 16 bytes per point. A time-series store keeps one open chunk per series in memory, encodes timestamps as delta-of-delta (a 10-second cadence compresses to a bit or two) and values as XOR against the previous value (a stable gauge compresses to almost nothing), reaching about 1.5 bytes per sample. Chunks are immutable once sealed, so storage is a log-structured sequence of blocks, read by time range, with an index from label values to series ids built per block. That structure is the reason both the write rate and the year-long queries are affordable.

How is the write path sharded, and what happens when an ingester dies?

By series id, consistently hashed across ingesters, so a series' samples are contiguous in one process and its chunk compresses well. Cardinality is the enemy: a label like request_id would create a new series per request and blow up memory, so ingest enforces per-tenant series limits and rejects high-cardinality labels with a clear error. An ingester's head block is reconstructed from its write-ahead log on restart, and samples are replicated to two or three ingesters (the log is consumed by a replication factor of 3) so a dead node loses nothing and queries read from any surviving replica, deduplicating by series and timestamp. Queries for the last two hours that hit ingesters are the hot path for alerts; blocks in object storage serve dashboards over longer ranges.

How do alerts stay under a minute and avoid double paging?

Evaluation interval 15 seconds plus scrape interval 10 seconds plus ingestion lag of a few seconds gives roughly 30 seconds from a bad sample to a condition being true; the "for 5 minutes" clause is a user choice that trades noise for speed. Evaluators are sharded by rule and replicated two ways; both replicas evaluate and both send notifications, and the notification layer deduplicates by alert fingerprint (hash of rule and labels) within a window, groups related alerts, and honours silences. A rule that depends on data that stopped arriving must fire "no data" rather than go quiet, which is the single most important property of a monitoring system: absence of samples is itself an alert.

How does the monitor survive the outage it is supposed to detect?

By sharing nothing with production: its own clusters, its own storage accounts, its own DNS, and alert delivery that does not go through the company's main messaging stack. A second small instance watches the first (a dead-man's switch: an alert that always fires and whose absence pages). Capacity is planned for the worst day, when every service is erroring and emitting more series than usual; per-tenant limits keep one noisy team from starving the rest.

6. What a strong answer sounds like

• Convert series and scrape interval into samples per second, then compression into bytes; the numbers justify the in-memory head and the block store.
• Explain the time-series data structure (per-series chunks, delta-of-delta, XOR) when asked why not a normal database; the interviewer in the reports pushed on exactly that.
• Shard by series id, name cardinality as the risk, and say how it is limited.
• Describe alerting as a pipeline with a latency budget and a "no data" rule.
• Expected follow-ups: histograms and percentiles (store buckets, aggregate then compute p99, never average p99s); multi-tenancy; cost of long retention and how rollups bound it.
• Common mistake: averaging latency percentiles across hosts, or designing alerting that goes silent when the pipeline breaks.

02Where this question was reported

• Exponent · Detailed report: collect several log/metric types, process in under a minute; follow-ups on the processing layer
• DesignGurus · 'Log Collection System'
• System Design Handbook · 'Metrics or Logging System'
• interviewkickstart.com · 'distributed logging system, such as the Google logging infrastructure'; 'log search system'
• PracHub · 'distributed log storage service', Hard
• PracHub · 'log management with auto-deletion'; 'prioritized log manager with eviction'
• scaleengineer.com · 'store and retrieve user activity logs for a global service'
• Hello Interview · 'Design a global VM monitoring system', commonly asked L5
• Hello Interview · 'Design a Server Health Monitoring System', commonly asked L6
• Hello Interview · Commonly asked Google EM
• PracHub · 'Design a Security Monitoring Framework', 1130 solves
• IGotAnOffer · EM low-level list (also in a Jan 2026 EM report surfaced by search)