Design server health monitoring and failure detection

Know within seconds which of a million machines is sick, without the monitor itself melting.

Problem 082 of 100Data pipelines and operationsL51 Nov 2026

Concepts this design leans on

Failure detection and health checks — Telling a slow machine from a dead one, quickly and without false alarms.
Presence and heartbeats — Knowing who is online and when someone silently vanished.
Time-series storage — Append-only metrics with compression, downsampling and retention.
Sharding and partitioning — Splitting data by key so each machine holds a slice it can handle.
Observability: metrics, logs and traces — The three signals that explain why a request was slow.

01The design, in the six-step framework

1. Requirements

Functional.

Every machine in a fleet of up to one million (physical hosts and VMs) reports whether it is alive and a handful of vital signs (CPU, memory, disk, network errors, service process up).
The system decides, per machine, healthy / degraded / dead, and tells schedulers and load balancers so work is moved away from sick machines.
Operators see fleet-wide health by cluster and rack and get an alert when failures cluster (a rack, a switch, a kernel version).
Out of scope: application metrics and dashboards in general (the metrics system), repair workflows beyond opening a ticket.

Non-functional.

Detection time: a dead machine is marked within 30 seconds; a degraded one within 2 minutes.
False positives under 0.1 percent per day per machine: marking healthy machines dead moves work for nothing and can cascade.
The monitor must not take down the network or itself at a million hosts; its traffic is under 1 percent of fleet bandwidth.
Availability 99.99 percent, independent of the fleet it monitors.

Back of the envelope

One million hosts sending a 200-byte heartbeat every 5 seconds is 200,000 messages per second and 40 MB/s: trivial bandwidth, non-trivial as a single receiver.

State: 10⁶ hosts × 1 KB of current state = 1 GB, which fits in memory on a few machines

Vital signs at 10 numbers per host every 10 seconds is 1 million samples per second into the time-series store; detection runs on the heartbeat stream alone, so it is cheap.

2. Core entities

Host: host_id, cluster, rack, hardware class, kernel version, expected state (in service, draining, maintenance).
Heartbeat: host_id, sequence, sent_at, boot_id, quick vitals.
Health verdict: host_id, state (healthy, degraded, suspect, dead), since, reason, confidence.
Probe result: host_id, prober, kind (TCP, HTTP, SSH), latency, outcome.
Incident: scope (rack, cluster, version), count, first_seen, correlated signals.

3. API

Agent → monitor: POST /heartbeat {host_id, seq, boot_id, vitals} every 5 seconds, over UDP or a long-lived gRPC stream; no response needed.
GET /hosts/{id}/health → verdict; and GET /clusters/{id}/health?state=dead → list, paged.
POST /hosts/{id}/maintenance {until} → suppresses verdicts during planned work.
Outbound: a host.state.changed event on a topic that schedulers and load balancers subscribe to; a webhook to the ticketing system for dead hosts.

4. High-level design

From a silent host to a moved workload:

Each host's agent sends a heartbeat every 5 seconds to the nearest gateway; gateways are spread over regions and reached by anycast so no single receiver sees a million senders.
Gateways append heartbeats to a stream partitioned by host id; the detector owning that partition keeps the host's last-seen time and sequence in memory.
If three consecutive heartbeats are missing (15 seconds), the detector marks the host suspect and asks probers in two other racks to reach it directly.
If probes fail too, the host is marked dead at about 25 seconds, the verdict is written to the state store and published on the state-change topic.
Schedulers drain work from the host, load balancers stop routing to it, and a ticket is opened with the reason and the last vitals.

5. Deep dives

How do we tell dead from slow, and why not just use a timeout?

A single timeout is either too short (false deaths during a network blip, GC pause or a saturated link) or too long (slow detection). The heartbeat gives a cheap, continuous signal; the active probe gives an independent second opinion from a different network path before anything drastic happens. The phi-accrual idea sharpens this: instead of a fixed timeout, keep the distribution of each host's inter-arrival times and compute how improbable the current silence is; a host that always heartbeats at exactly 5.0 seconds is suspect after 7, one on a jittery link is not suspect until 12. Sequence numbers and boot ids catch two more cases: a host that rebooted (new boot id, mark it as recovered-after-reboot, not healthy throughout) and a host whose heartbeats arrive late but in order (the network, not the host, is the problem).

What keeps the monitor from melting at a million hosts?

Fan-in is the risk: a million senders to one endpoint is a self-inflicted attack. Regional gateways with anycast spread the receiving; the stream is partitioned so detectors scale horizontally; per-host state is a few hundred bytes, so 100 detectors hold the whole fleet with room to spare. The gateways do not acknowledge heartbeats, so a sender never blocks on the monitor. For the reverse direction, probers are rate limited per target and per prober. The state store is written only on state changes, which are rare; the common case of "still healthy" never touches a database. And the heartbeat rate is the knob: 5 seconds gives 30-second detection; halving it doubles the traffic for a marginal gain.

Approach	Detection time	Load	False positives
Central pinger polls every host	Poll period × fleet ÷ parallelism; minutes	Pinger is the bottleneck	Low
Push heartbeats, fixed timeout	Timeout, about 30 s	Spread by gateways	High during network blips
Heartbeats + adaptive timeout + probe confirmation	20 to 30 s	Slightly more, bounded	Low; two independent signals
Gossip between hosts	Seconds	Spread over the fleet	Low, but harder to reason about and to operate

How do we spot a rack or a kernel version failing, not just one host?

Every verdict carries the host's rack, switch, cluster and software version. A correlation job over the state-change topic counts dead and degraded hosts per group in a sliding 5-minute window and compares against the group's baseline; a rack going from 0 to 20 dead in a minute is a switch, not twenty disks, and the right action is one ticket for the network team and a pause on moving those workloads elsewhere (they will come back). This is also where false positives are caught: if a whole region goes quiet at once, the far more likely explanation is that the monitor lost connectivity to it, so the detector holds verdicts for a region whose heartbeat volume drops by more than a threshold and raises a monitor-side alert instead.

What happens when the monitor itself fails?

Detectors are replicated: two detectors consume each partition and compute the same verdict; a verdict is published when the first replica reaches it and deduplicated downstream by (host, state, since). Detector state is rebuilt from the stream on restart, which holds the last few minutes of heartbeats, so a restarted detector is caught up within one heartbeat interval. The monitor runs in its own clusters, with a second tiny instance watching it; a verdict that is older than a few minutes for the entire fleet is itself an alert.

6. What a strong answer sounds like

Do the fan-in arithmetic and show that bandwidth is trivial but the single receiver is not, which motivates gateways and partitioning.
Separate the signal (heartbeats) from the confirmation (active probes) and explain why both are needed to keep false positives low.
Describe detection time as a budget: heartbeat interval × missed count + probe time.
Talk about the whole-region-silent case; it is the failure mode that distinguishes people who have run one of these.
Expected follow-ups: how to monitor services on a host versus the host; how to handle planned maintenance; how to avoid a thundering herd when a dead host comes back and everything reschedules at once.
Common mistake: a central poller, which does not scale, or a timeout with no second signal, which pages all night.

02Where this question was reported

Hello Interview · 'Design a global VM monitoring system', commonly asked L5
Hello Interview · 'Design a Server Health Monitoring System', commonly asked L6
Hello Interview · Commonly asked Google EM
interviewkickstart.com · 'detect node failures in a cluster of 10,000 nodes'
PracHub · 'Design a Security Monitoring Framework', 1130 solves
interviewkickstart.com · 'Detect node failures in a cluster of 10,000 nodes'.
grokkingsystemdesigninterview.com · L6 planet-scale monitoring platform.
1point3acres · Per search snippet (page 403): Google onsite system design on AWS EC2 metrics.